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ABSTRACT 



This manual, which is intended for practitioners involved in 
the Asia and the Pacific Programme of Education for All, explains the basic 
concepts and procedures involved in monitoring and evaluating literacy and 
continuing education programs. The following are among the topics discussed 
in the manual's eight chapters: (1) understanding monitoring concepts and the 

process of developing a program monitoring plan (objectives, characteristics, 
tasks of monitoring systems) ; (2) developing indicators for program 

monitoring (types of indicators, convergence of indicators with monitoring 
tools) ; (3) designing methodologies in monitoring (types and characteristics 

of monitoring tools, data collection and analysis methods) ; (4) reporting and 

disseminating monitoring results (educational management information 
systems) ; (5) understanding evaluation concepts and the process of developing 

an evaluation plan (need for and functions of evaluation) ; (6) identifying 

the four major areas of evaluation; (7) developing the evaluation design and 
managing the evaluation work (sampling methods, evaluation roles and 
responsibilities, staff orientation and training) ; and (8) reporting and 
using evaluation results (reporting formats, the target audiences of 
evaluation reports, factors affecting use of evaluation results). The 
document contains 19 references and 22 tables/figures. Appended are sample 
forms, sample evaluation tools, and a list of participants. (MN) 
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INTRODUCTION 



In the pursuance of UNESCO’s ultimate goal, which is to promote primary education, 
literacy and continuing education for all children, youths and adults, the Asia and the 
Pacific Programme of Education for All (APPEAL) has adopted various strategies to 
reach the ultimate beneficiaries of education. Its priority programme areas are: 

tt Reaching the unreached, the underserved and the disadvantaged population groups in 
rural and urban areas, with a particular emphasis on girls and women; 

0 Promoting community participation and ownership; and 

0 Improving the relevance and quality of basic education and enhancing the achieve- 
ments of all children, youth and adults. 

In order to achieve this ultimate goal, the participating Member States of APPEAL have 
implemented educational programmes at the national, provincial and local levels. At these 
various levels of implementation, planners and administrators of these programmes see the 
strong need for improving their monitoring activities and systems. Further, they feel that 
programme monitors and evaluators need more user-friendly and simple monitoring/ 
evaluation tools that will help them conduct their work. 

The Practitioners ’ Manual on Monitoring and Evaluation of Literacy and Continuing 
Education Programmes is APPEAL’S response to this need for further strengthening of 
the monitoring and evaluation activities of the Member States. It is designed to 
complement the earlier efforts of APPEAL in providing information and educational 
materials on programme monitoring and evaluation of literacy and continuing education 
programmes. Most notable of these products are the APPEAL Training Materials for 
Literacy Personnel (ATLP), the APPEAL Training Materials for Continuing 
Education Personnel (ATLP-CE) and the APPEAL Manual for Planning and 
Management (AMPM). The basic distinctions between this manual and the ATLP and 
the AMPM stem from the contextual framework and the intended users of the materials. 
While the ATLP and the AMPM provide overview and general orientation materials 
useful to managers and planners, this manual is designed to give operations-oriented advice 
that can be used at the field level, especially by community-based service providers. Thus, 
the primary target users of this manual are field-based programme administrators who 
need to monitor the day-to-day implementation of literacy and continuing education 
programmes and projects, as well as programme evaluators. It gives them concrete and 
specific examples as well as step-by-step processes in the preparation and actual 
implementation of monitoring and evaluation activities. 

The following topics are included in this manual: 

Definitions and concepts of monitoring and evaluation; 

Distinctions between monitoring and evaluation; 

Development and preparation of monitoring and evaluation plans; 
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• Development of monitoring and evaluation indicators; 

• Designing tools and managing monitoring and evaluation; 

• Training as a capacity-building strategy in monitoring and evaluation; and 

• Reporting, dissemination and use of monitoring and evaluation findings. 

The manual is divided into two main parts. Part One covers monitoring concerns while 
Part Two focuses on evaluation. In each part, the topics listed above are comprehensively 
discussed. Whenever possible, related issues are linked together in the discussions. Also 
presented are simple and easy-to-follow examples and procedures in the planning and 
actual performance of monitoring and evaluation work. They can serve as models that 
may be adopted or adapted by field programme administrators and evaluators. 

The manual is the major output of the UNESCO Technical Working Group Meeting 
on Monitoring and Evaluation of Literacy and Continuing Education organized 
by APPEAL and the SEAMEO INNOTECH on 4-10 May, 1998 in Quezon City, 
Philippines. Participants were evaluation experts from Bangladesh, China, India, 
Malaysia, Philippines and Thailand (Appendix IV). Before writing the manual, they held a 
series of workshops and devised a common framework that identified the concerns of 
monitoring and evaluation reflecting the realities of the region. Although they individually 
worked on specific parts of the manual, they reviewed and assessed the content as a group. 
This group effort is an important element in the preparation of the manual because it 
provides a regional dimension to the framework, the concepts discussed and the 
monitoring/evaluation models presented 
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PART I : MONITORING 




Chapter One 



UNDERSTANDING MONITORING CONCEPTS AND 
DEVELOPING A PROGRAMME MONITORING PLAN 



This chapter aims to enable programme planners and 
practitioners of literacy and continuing education 
programmes to: 

* define monitoring and set it in the overall programme 
context; 

* differentiate monitoring from evaluation; 

* understand the benefits that can be derived from 
conducting regular monitoring; 

* discuss the objectives of programme monitoring and 
identify the features of good monitoring objectives; 

* discuss the characteristics of a good monitoring 
system; 

* understand the tasks of a programme monitor; 

* descii.be how monitoring plans are prepared; and 

* discuss the major steps in preparing a monitoring 

plan. 
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1.1 Programme Monitoring Defined 

The following definitions provide appropriate descriptions of the concepts of programme 
monitoring (SEAMEO INNOTECH, 1997): 

♦ Programme monitoring is a system of (1) documenting all planned efforts, (2) 
processing the raw data, and (3) disseminating the processed information to higher 
authorities like the management and planning officers. 

♦ Programme monitoring is a systematic framework for collecting and analyzing 
information concerning all events related to the implementation of the project in order 
to improve its management. 

♦ Monitoring is the process of gathering and analyzing data on the progress of ongoing 
programme activities and providing feedback information to programme managers for 
taking corrective actions. 

♦ Monitoring is a responsibility of management. It refers to the planned collection of 
data on the programme /project implementation, processing these data, and 
communicating processed information promptly to its users so that it can help further 
improve programme implementation. 

♦ Monitoring is the systematic attempt to measure: (1) the extent to which a 

programme reaches its intended target population (programme coverage), and (2) the 
extent to which the services being delivered match what was intended to be delivered 
(programme process). 

The common features of monitoring in the above definitions and descriptions are: (1) it is 
a system which follows a well-designed process, and (2) its primary purpose is to generate 
information that can be used to improve programme implementation. 

1.2 Monitoring as a Management Function 

Monitoring programme implementation is a major function of managers, planners, field 
administrators and other practitioners in literacy and continuing education programmes. It 
is a mechanism that provides information on programme progress at various levels and 
stages of implementation. The manager needs field-based information for making 
appropriate decisions about programme directions and operations strategies. The planner 
needs data on the strengths and weaknesses of the programme plan in order to identity 
possible gaps between the planned and the actual activities implemented at the field level. 
The programme administrator's interest in monitoring lies in the area of determining 
whether he or she is reaching clients and intended beneficiaries. 

A well-designed and carefully-scheduled programme monitoring follows a systematic 
framework for collecting and analyzing information on implementation activities with the 
end view of improving the management and operations at the various stages of the 
programme. It can provide information to policy makers and programme management so 
that timely and appropriate decisions can be made, thus helping ensure the success of the 
programme. 
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1.3 The Conceptual Interface between Monitoring and Evaluation 

There is often a tendency to equate monitoring with evaluation and to use the two words 
interchangeably and synonymously. In actuality, these are two different programme 
components and their conceptual interface is basically in terms of the time frame when 
they may occur as parallel activities. 

Figure 1.1 shows the conceptual interface between monitoring and evaluation. Also 
presented are basic questions that these two programme components attempt to answer. 

In programme monitoring, the efforts are mainly aimed at ensuring that the programme is 
functioning as planned. Unlike evaluation, which aims to judge programme performance, 
the overall aim of monitoring is to ensure that the programme activities are implemented as 
planned and whether the factors that lead to success are present in the field. The major 
monitoring function is to provide profiles of activities at the implementation level. 
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Figure 1.1 


Conceptual Interface between Monitoring and 
Evaluation and Basic Objectives and Questions 






Programme Start Programme 

Termination 


Planning Phase 


Implementation Phase 


Post-Programme Phase 



A Monitoring ► 

A Evaluation ► 



Monitoring: Basic Questions 

1. Are project activities being 
implemented according to plan? 

2. Who is benefiting from the project? 

3. Are the expected outcomes/outputs 
being developed? 

4. Is the amount of benefits being 
delivered the right amount? 

5. What changes, if any, should be 
made to the programme? Why? 



Evaluation: Basic Questions 

1. To what degree has the objective 
been attained over time? 

2. Is the project cost effective? 

3. What impact has the project had 
upon the target clientele? 

4. What decision should be taken on 
the programme? 



We MONITOR to "PERFECT" implementation 
We EVALUATE to "JUDGE" the performance 



* of the programme 



Source: SEAMEO-INNOTECH Course Handouts on Planning and Management of Literacy and 

Continuing Education Programme, 1996-1997. 
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Evaluation, on the other hand, aims to examine the project in its entirety - the context, 
input, process, output and outcome - and make recommendations that may lead to the 
revision of the programme design or replacing it entirely. It may also recommend 
changes in the ftiture course of action for the programme. 

The basic difference between monitoring and evaluation, therefore, lies in their purposes - 

the major purpose of monitoring is to improve and perfect the implementation of the 
programme and point out gaps so that the programme manager can address them, 
while that of evaluation is to assess and judge programme performance. 

In terms of programme time frames, both monitoring and evaluation activities are planned 
before the actual programme implementation. Monitoring work commences simul- 
taneously with the start of the programme while evaluation work is undertaken at regular 
intervals sometime after the programme begins, when it ends and/or sometime after its 
completion. 

1.4 The Importance and Benefits of Programme Monitoring 

Programme monitoring is important because of the following reasons: 

o 

♦ Monitoring information serves as a basis for modifying or revising the ongoing 
activities of the programme. Insufficient or totally absent information critical to the 
success of the programme is a sign of the ineffectiveness of the monitoring system put 
in place and made operational during the programme period. 

♦ Monitoring is a necessary component in preparing for impact assessment. Experiences 
in project monitoring have shown that the failure of programmes is often due to weak 
or incomplete implementation of programme interventions rather than the ineffec- 
tiveness of the interventions themselves. 

♦ Programmes in literacy and continuing education continually seek answers to questions 
such as: Who is benefiting from the programme? How are services being delivered? 
Are the beneficiaries actually learning what they are supposed to learn? These 
questions can only be answered if there is an existing monitoring system for the 
programme. 

Programme monitoring provides the following benefits to management, the planners and 
field administrators: 

♦ They can measure the progress of implementation against project plans and budget 
allocations for effective management and control 

♦ They can identify problems, weaknesses, gaps, bottlenecks, issues and potential 
sources of delays in order to devise corrective actions that will improve programme 
implementation . 
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1.5 Objectives of Programme Monitoring 

The objectives of programme monitoring are generally derived from the questions asked in 
Figure 1.1. They are the issues that are normally addressed, by programme managers and 
planners, as well as field administrators: 

* r 

Some sample programme monitoring objectives are as follows: 

♦ Identify the actual status of the long-term sustainability of the programme in order to 
determine if scheduled activities and expected outputs are being implemented/achieved 
as planned. 

♦ Discover gaps and deficiencies in programme implementation as well as current and 
potential issues/problems which need to be addressed. 

♦ Identify factors that may affect the long-term sustainability of the programme. 

♦ Highlight significant features that may serve as insights for the planning and 
implementation of future similar projects. 

♦ Identify opportunities and strengths that can be tapped to ensure successful 
programme management and implementation. 

♦ Recommend policy options for effective and efficient programme management and 
implementation 

♦ Document initial success stories that may be useful for social mobilization and 
advocacy, or for replication in related programmes and projects. 

♦ Immediately inform programme management and administrators of the status of 
programme implementation so that they can apply remedial measures to solve 
problems. 

♦ Disseminate the results of the review, monitoring and evaluation work to the 
programme managers, planners, donors and other sponsors. 

Programme monitoring objectives should be: 

•Specific 
Measurable 
A chievab le/attainab le: 

/?esult-oriented, 

71me-bound, as set by the programme 

Objectives that are specific help to clarify goals, provide direction to the activities, guide 
the selection of data and provide a basis for assessing the overall value of the evaluation 
Measurable objectives help provide outcomes that are observable and demonstrable. An 
achievable objective should be attainable on the basis of the available financial and 
material resources, and the technical demands of the methods to be employed, as well as 



1 . The Literacy Coordinating Council (LCC) of the Philippines, Mid-term Review of the Comprehensive 
Education and Community Development Programme (CECDP) of Valencia, Negros Oriental, 
1997. 
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staff capabilities. Lastly, objectives should be time-bound and should be achieved within a 
specific time frame 

1.6 Characteristics of a Good Monitoring System 

In order to ensure the chances of success of the programme, monitoring is needed at 
various stages of the programme cycle from the inception of the project up to the 
completion of the final activities. A good monitoring system has the following 
characteristics; . 

♦ It should provide periodic and timely feedback on physical and substantive 
programme accomplishments as well. as financial status 

♦ It should identify problems that require solutions and action . Problems may be 
specific to the programme or may be institutional in nature (e.g., changes in 
procurement procedures and rules governing overtime pay) 

♦ It should be relatively simple and should not burden the programme monitors , but 
comprehensive enough to incorporate the basic information, required by programme 
management. The use of complicated forms requiring a large volume of data not only 
creates confusion but will reduce the enthusiasm of those who are directly responsible 
for data collection and monitoring 

♦ It should be a continuing exercise from the start-up phase toAhe project completion 
stage Monitoring seeks to provide a profile of real-world situations, to compare what 
actually happens with what has been planned or expected For effective problem 
solving, monitoring should ensure timeliness of information and feedback. ’ 

♦ It should take into consideration the different areas of accountability , namely; 

Coverage accountability . Are the persons being served by the programme, really the 
intended beneficiaries? Are there beneficiaries who are not reached by the 
programme? Are there persons who are not intended beneficiaries being served by the 
programme? :f 

Service delivery accountability . Are services being delivered in the proper amounts? 
Are the treatments and interventions provided really the intended programme services? 

Fiscal accountability . Are funds being used properly? Are expenditures properly 
documented? Are funds used with the limits set by the budget? 

Legal accountability \ . Are the relevant statutes and rules being observed by the 
programme? 

1.7 Tasks of the Programme Monitor 

The programme monitor is normally a regular member of the programme team He or she 
is primarily responsible for the collection of information that can be used to improve the 
programme as it develops During the programme planning stage, the project monitor 
undertakes activities resulting in the necessary information that will help planners adjust 
the programme design to the realities of the field ' 
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The major tasks of the programme monitor are: 

♦ To determine, together with programme planners and implementation staff, what kinds 
of information about the programme will be collected and shared, and what decisions 
will be made based on the information generated from various sources. 

♦ To ensure that the programme goals and objectives as well as the major strategies for 
implementation have been carefully thought out and clearly stated in programme 
documents. 

♦ To collect data at programme sites about how the programme is actually operating in 
the field. 

♦ To report this information clearly and to help the staff plan necessary programme 
modification. 

Based on the above tasks, the monitoring agenda consists of the following activities: 

♦ Prepare a clear description of the programme monitoring plan, including its goals and 
objectives, activities and schedules. 

♦ Define the scope of the data-gathering activities and the responsibilities of the 
concerned monitors. 

♦ Conduct the necessary orientation and training of monitoring staff in order to 
adequately prepare them for their tasks. 

♦ Monitor and document programme implementation based on the scope of the data- 
gathering activities. Report and confer with planners and programme administrators on 
the findings of the monitoring activities. 

1.8 Preparing a Monitoring Plan 

The development and preparation of a programme monitoring plan are normally done 
before the start of the implementation phase They are major tasks of the monitoring staff 
of the programme. The programme managers and planners are consulted in order to arrive 
at a common understanding of the objectives, nature and scope of the monitoring system. 
The schedule of the monitoring activities is also reviewed and discussed with them so that 
they know when to expect data from the field and subsequently make the necessary 
analysis of the information gathered by the programme monitors. 

After review and consultation, a plan for the monitoring system is developed. The major 
elements of the plan are: 

♦ Objectives 

♦ Indicators 

♦ Programme activities to be monitored 

♦ Scheduled implementation period 

♦ Expected activity outputs and dates of completion 

♦ Critical data to be gathered 
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♦ Data-gathering tools and/or techniques 

♦ Monitors responsible for the actual monitoring 

♦ Monitoring dates 

♦ Budget 

The first two elements, objectives and indicators, are developed before the actual 
programme implementation and may be taken from the programme monitoring plan. The 
other eight elements are the major items to be accomplished during the actual monitoring 
work. 

These elements may be plotted on a chart, as follows: 

Monitoring Objective(s):^ _ 

Monitoring Indicator(s): 




1.9 The Process of Developing a Monitoring Plan 

Step One: Develop the monitoring objectives. These objectives are generally based on 

the monitoring questions and concerns identified in Figure 1.1 of this 
chapter. They are also normally contained in the overall monitoring plan of 
the programme. 

Step Two: Specify the programme monitoring indicators. These indicators will serve 

as the basis for developing the appropriate monitoring designs and 
methodologies. These indicators are likewise normally provided in the 
overall monitoring plan of the programme. 

Step Three: Identify the programme activities to be monitored. For easy reference, a 
simple Gantt Chart may be prepared (Figure 1.2). 
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Figure 1.2 Sample Gantt Chart for Programme Monitoring 
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Step Four: 
Step Five: 



Step Six: 
Step Seven: 

Step Eight: 

Step Nine: 
Step Ten: 
Step Eleven: 



O 




Indicate the expected outputs from each of the activities and the 
dates when these outputs shall appear. This step should be care- 
fully done with the programme planners to ensure accurate monitoring. 

Prepare a list of the activities to be monitored. This list should 
also specify the following: 

♦ Actual events at monitoring time; 

♦ Participating programme beneficiaries and/or administrators; 

♦ Implementation schedule; 

♦ Expected outcome/output of each activity; 

♦ Resources/inputs being used; and 

♦ Person(s) in charge of the activity. 

Take note of the date(s) of implementation of the activity(ies) to be 
monitored. 

Identify the important data to be gathered about the activity(ies). These 
are data which the programme management needs in order to direct and 
control programme implementation. 

Choose the appropriate dates when the data identified in Step Five will 
be gathered. The programme management should participate in the 
selection of these dates so that they will know when to expect data from 
the field. 

Prepare the monitoring methodologies and instruments. The monitoring 
plan should describe the data-gathering techniques and instruments to be 
used. 

Prepare the monitoring budget. All anticipated expenditures to be 
incurred should be identified, including provisions for a possible increase 
in costs. 

Prepare and validate the plan prior to actual monitoring. The 
programme monitors and/or programme staff assigned to write the plan 
for the programme monitoring system should frequently consult with 
management and planning officers throughout the writing and plan 
preparation stage. 
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Chapter Two 



DEVELOPMENT OF INDICATORS FOR 
PROGRAMME MONITORING 



Tills chapter aims to enable programme planners 
and practitioners of literacy and continuing 
education programmes to: 

* identify various types of indicators; 

* describe the rationale for the development of 
indicators; and 

* explain the process of developing indicators. 
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2.1 Monitoring Indicator Defined 

An indicator is a variable that reflects the efficient and effective performance of a 
programme. Indicators provide a standard against which to measure changes brought 
about by programme activities. They are determined^ on the basis of programme 
objectives. For example, in ^monitoring the process v;pf literacy programme 
implementation, you can look at management style, learning contact hours and 
instructional supervision, among other indicators. 

2.2 The Need for Developing Indicators in Programme Monitoring 

In order to help programme managers and operations staff at the field Jevel in monitoring 
the activities of literacy and continuing education programmes, a set of indicators, have 
to be identified and developed. Indicators are useful tools in documenting activities and 
measuring changes in the programme over time. Wheh' indicators are developed on a 
regular basis, they can help identify possible changes that need to be made in the 
programme structure, strategies Or implementation scheme. For example, socio- 
economic indicators can be used to measure the changes in the socio-economic status of 
a village, group or family If there are no clear and measurable indicators, it would not 
be possible to measure the progress of a literacy programme; likewise, it would be 
impossible to collect appropriate and precise data from the project site. 

2.3 Characteristics of a Good Indicator 

A good indicator has the following characteristics 1 : 

♦ It is measurable. 

♦ It describes central features of the programme implementation. 

♦ It links to the basic goals of the programme by providing a measure of progress or 
change. 

♦ It is statistically valid and reliable. 

♦ It can be readily observed and measured in terms of time, expertise and cost. 

♦ It is comprehensive and meaningful. 

♦ It is understandable to all major interested users. 

♦ It identifies trends over time 

♦ It is applicable or useful at different levels (national, local, and grassroots). 

♦ It is relevant to education policy. 



1. Based on the UNESCO PROAP Educational Management Information System Training Manual , 
Bangkok, 1998. 
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2.4 Types of Indicators 

There are various types of monitoring indicators for literacy and continuing education 
programmes and their categorization depends on their scope and purposes. 

Qualitative indicators are intended to measure the quality of the input, process and 
output of the programme. They measure performance relative to some given standards 
and norms. The term ‘quality” can mean different things depending on the context. 
Here the term refers to a perceived improvement in the implementation of the literacy 
programme. Examples of qualitative indicators are: 

♦ leamer/teacher ratio by level; 

♦ classrooms in good condition; 

♦ absenteeism (learners, teachers); and 

♦ use of multimedia teaching aids. 

Quantitative indicators statistically measure the amount or value of inputs or resources 
available. The “quantity” reflects a numerical condition such as the number of learners, 
teachers, costs, facilities or textbooks at a specified time. Examples of quantitative 
indicators can include: 

♦ learner enrolment percentage, including females; 

♦ costs/expenditure per learner by level and urban/rural location; and 

♦ textbooks by level and urban/rural location. 

Input indicators are determinants subject to policy manipulation, e.g., the 
characteristics of learners, teachers, curriculum, textbooks, other instructional materials, 
facilities, equipment, learner capacity for learning and other resources. 

Process indicators are determinants that reflect forms of interaction between teachers, 
learners, administrators, materials and technology. Process indicators refer to the 
procedures or techniques that determine the transition of inputs into outputs, and are thus 
important for evaluation. 

Output indicators are results or changes readily observable upon completion of a level 
of education. These indicators mainly refer to student attainment and student 
achievement, specifically test scores that reflect value added. 

Equity indicators are used to measure the degree to which expenditures for education 
are provided for the population regardless of economic status, place of residence and 
intellectual capability. These also measure equality of access not only to physical 
facilities such as schools or learning centres but also to good quality education. 

Efficiency indicators are . used to monitor the attainment of one of the programme’s 
results at the least possible cost. Cost is basically the expenditure associated with the 
use of resources such as personnel or equipment. Examples of efficiency indicators are 
promotion rates, repetition rates, drop-out rates, average study time, and rate of facilities 
use. 
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Outcome indicators are results and effects on individuals and society as a whole that are 
evident over time as a consequence of education outputs within the socio-economic 
context. These education outcomes are effects more distant in time after completing 
education, and are usually more dispersed in occurrence than education outputs. The 
main outcome indicators are: 



♦ degree to which acquired learning and skills (e.g., literacy and numeracy) have been 
retained several years later; 

♦ admission to further education and training; and 

♦ achievement in subsequent education and training. 

Learning-related indicators are focused on a learner’s condition and needs. The 
indicators include recruitment rate, achievement/performance level of learner, and 
learner’s attitudes and motivation. 

Instruction-related indicators include instructor’s performance/achievement level, 
his/her teaching methods and strategies, sufficient instructional materials and other 
factors related to instruction. 

Resour ce/facilities-r elated indicators include availability of learning centres, teaching 
equipment at learning centres, learning materials, and relevance of the curriculum. 

Socio-economic/environment-related indicators are concerned with the level of 
community involvement, participation of government and non-government organizations, 
and the standard of living of the participants in the programme. 



2.5 Development of Indicators 



The following steps are useful in developing measurable and effective indicators 



Step One: 



Step Two: 



Step Three: 



Step Four: 



Study the objectives and expected outcomes of the literacy programme. 
The best strategy is to identify and clearly define the objectives of a 
literacy programme and break down these objectives into concrete small 
targets or elements. These objectives should include the expected 
outcomes of the literacy programme. 

Specify activities for the planned input, process and output. For 
example, input consists of personnel, money and materials. Activities 
for materials would include needs assessment, compiling and 
dissemination. 

Identify parameters or limits for measurement of performance. The 
parameters, should clearly reflect the main characteristics of 
performance and be measurable. Once the parameters are identified, the 
important task is to determine ways to quantify them. 

Check whether the identified indicators match with the contextual needs 
related to learning, instruction, facilities, and socio-economic 
environment. 
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Step Five: Segregate the appropriate indicators at different functional levels, for 

example, village, district or national. 

Step Six: Develop a matrix of indicators. It should show the logical and internal 

relationship between indicators and programme objectives at different 
functional levels. The following table gives an example of such a 
matrix: 



Table 2.1 Matrix Showing the Relationship between 
Indicators and Progralmme Objectives 



Programme 

Objectives 


! Suggested Indicators 


Means of Measurement 


| Provide basic literacy 
| to all illiterate adults 
I 15-45 years old 


■ Enrollment of illiterates 

■ Completion rate 

■ Rate of drop-out 

■ Etc. 


■ Percentage of total 
illiterates in the area 

■ Percentage of enrolled 
learners 


j Improve the skills of 
| the learners in 
| community water 
i management 


■ Skills for analysis of watgr 

j problems in the community 

■ Competency in problem 
solving through community 

| participation 

■ Knowledge about low-cost 

i water management system 


■ Percentage of learners 
acquiring the skills 

■ Ways of 
demonstrating 
problem solving skills 

■ Areas of increased 
knowledge 



2.6 Some Guidelines for Developing Indicators 

The planners of a programme monitoring system should consider many aspects to make 

the indicators environmentally sound and cost-effective. The following guidelines may 

be useful: 

♦ Check whether the indicators have the characteristics of good indicators (see 
Section 2.3). 

♦ Be very selective in identifying indicators and keep the list of indicators as short as 
possible. Avoid a very long list of indicators. Confine the list to those essential for 
the information required. 

♦ Consider the availability of human, financial and technical resources in the collection 
and processing of information. Management of large-scale information demands an 
increased budget for the collection and processing of data. It also requires additional 
skilled manpower. 

♦ Keep in mind how the information to be collected will be used. It is better to avoid 
collecting information that won’t be used immediately. (Outdated information is of 
little use for monitoring the progress of any project. 
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♦ Start with a small number of indicators and keep options for incremental expansion. 
It is easy to handle a smaller number of indicators at the beginning stage of any 
project. Gradually more and more indicators may be added depending upon the 
capacity of the project management and the need for more information. 

Focus on clearly-defined information needs. If the information needs are not specifically 
identified, there is the risk of unclear or inaccurate responses, which ultimately 
jeopardizes the very objectives of monitoring. Selection of indicators with specific 
information needs also helps in keeping the list of indicators shorter. 

Create an opportunity for the participation of stakeholders in identifying the indicators 
and their use. Participation of learners, community members and literacy personnel from 
various levels in designing the monitoring system helps in the development of a needs- 
based system. Further, it increases their sense of ownership of the programme and 
ultimately facilitates the collection of good quality data at their level. 

There should be periodic review and updating of indicators. Indicators developed at the 
initial stages of the programme should not be considered as permanent monitoring 
indicators. With changes in the socio-economic environment or implementation system 
of the programme, the indicators might need to be changed or revised. Thus it is better 
to make provisions for a periodic°review of indicators. 

2.7 Sample Indicators for Monitoring Literacy Programmes 

For monitoring a literacy programme, indicators are developed for the various levels of 
programme implementation. However, not all indicators may be used by all programme 
staff. A top-level manager may not necessarily require the information essentially 
required by a village supervisor. For example, a supervisor would continuously require 
information regarding the use of supplementary teaching aids to ensure adequate input 
for learning but the top manager of the programme doesn’t need to use such information 
on- an ongoing basis. 

In the following table, certain indicators have been listed according to their use at 
various levels - from the learning centre to the national level. This is not an exhaustive 
list and the indicators shown at each level are not exclusive for that level. Programme 
administrators at various levels may simultaneously require more or less the same 
information; thus similar indicators are mentioned repeatedly in Table 2.2 showing the 
various levels of programme implementation. 




28 



Development of Indicators for Programme Monitoring O 19 



Table 2.2 Indicators at Different Levels: A Typology 







| Functioning 
Levels 


Indicators 


j Learning centre 
j level 


Percentage of targeted learners enrolled 
Percentage of women/disadvantaged people enrolled 
Percentage of average attendance by gender/income 

Percentage of achievement of learners according to literacy levels and functional 
skills 

Percentage of learners who use literacy skills and frequency of use 
Percentage of learners who drop out and reasons (by gender/income) 

Percentage of learners who have access to educational materials 
Percentage of centres visited by community leaders/extension agency personnel 
Percentage of learners satisfied with the contents of the curriculum 
Percentage of family (of learners) where changes are observed as a result of 
literacy and types of changes ‘ 

Percentage of graduates who enter post-literacy programmes 


j Village/ 
j community level 


Percentage of learners who completed the course 
Percentage of centres with learner participation in classroom activities 
Percentage of centres where planned supplies are available with comments on 
quality and use of supplies 

Percentage of teachers provided training and types of training 




Level of teacher performance 

Percentage of budgeted funds available and used 

Percentage of centres with good learning environment (physical/contextual) . 
Number of learning centres established 
Average teacher/leamer ratio 

Number of organizations (GO and NGO) implementing literacy programmes with 
coverage of learners 


; District level 


Percentage of centres visited by supervisors and issues covered during visit 
Percentage of learning materials actually delivered/supplied 
Percentage of literacy personnel provided training with levels and types of training 
Level of teacher performance 

Percentage of supervisors at a certain level of performance 
Per capita cost 


j Province level 


Percentage of target learners who enrolled 
Percentage of women/disadvantaged people who enrolled 
Percentage of learners who completed the course 
Percentage of learners who dropped out (by gender/income) 

Percentage of budgeted funds available and used 
Number of learning centres established (by district) 

Number of organizations (GO & NGO) implementing literacy programmes, with 
clients and learners 
Per capita cost 


National level 


Percentage of target learners who enrolled 
Percentage of women/disadvantaged people who enrolled 
Percentage of learners who completed the course 
Percentage of learners who dropped out (by gender/income) 
Percentage of budgeted funds available and used 
Per capita cost 







BEST COPY AVAILABLE 



20 O Practitioners' Manual on Monitoring and Evaluation 



2.8 Some Indicators for Monitoring Continuing Education 
Programmes 

Continuing Education (CE) programmes are essentially follow-ups to literacy 
programmes. They aim to consolidate, reinforce and enhance the acquired knowledge, 
skills and competencies of neoliterates. These programmes are also designed to meet the 
functional needs of the people. Since these needs vary, there can be a variety of 
continuing education programmes to cater to different local needs. Taking into account 
the diversified client needs, UNESCO's ATLP-CE classifies continuing education 
programmes into six types: post-literacy programmes, income generation programmes, 
equivalency programmes, quality of life improvement programmes, individual interest 
promotion programmes and future oriented programmes. 

Like literacy programmes, continuing education programmes need to be monitored with 
a specific set of indicators. Monitoring these programmes is more important because of 
the diversified nature of their activities and concerns. Thus indicators need to be 
developed according to the objectives of the programmes. There cannot be a single set 
of indicators for monitoring all types of CE programmes. For example, indicators for 
monitoring post-literacy programmes cannot be used in monitoring income generating 
programmes. However, the basic steps for developing indicators for literacy 
programmes (see Section 2.5) can be used for developing indicators for continuing 
education programmes. 

The following are sample indicators for monitoring post-literacy programmes: 

Sample indicators for monitoring post-literacy programmes* 

• Number of community learning centres (CLCs) established with information 
| on type of building, furniture, recruitment of community workers and their 
!j o training, formation of committee 

;; ; 

|| • Number of enrolled members in CLC and percentage of learners who 

| graduated from literacy programmes 

|| • Percentage of CLCs provided with information, education and commu- jj 

|| nication (IEC) materials, with types of materials and quantities supplied 

jj • Percentage of member attendance at CLCs provided with IEC materials, 

j! including pattern of use, type of materials used and frequency of attendance 

jj • Number of meetings of CLC management committee, percentage of 

jj attendance for committee members and issues discussed 

j • Resources mobilized locally for the operation of CLCs 

jj • Important events/activities organized by CLCs and their frequency 

|j • Evidence showing benefit of CLCs at the learner and community levels 

|j • Percentage of CLC members with improved educational competencies and jj 

j specific areas of improvement 

[ * Adapted from the monitoring system of the Dhaka Ahsania Mission, Bangladesh. 
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Sample Indicators for Income Generation Programmes* 

• Types of training courses organized for imparting skills towards income 
generation and number of persons covered 

• Percentage of literacy graduates covered under Income Generation 
Programmes (IGPs) 

• Number of persons trained with entrepreneurship development, marketing 

skills, accounts keeping ; ■ 

• Number of persons linked up with other rural development agencies and type 
of agencies 

• Rate of participation of women in IGPs 

• Nature of scope of activities available for further enhancing of competencies, 
related to income generation 

• Rate of increase in income of the participants 

• Based on ATLP-CE . .. 



Sample Indicators for Quality of Life Improvement Programmes* 

• Extent to which the Programmes meet biological, social, economic, humanist 
and environmental needs 

• Percentage of resources that are mobilised locally 

• Percentage of participants who have access to basic services and information 

technology ; ' 

• Rate of participation of literacy graduates in the promotion of socio-cultural 
values 

• Nature of steps taken for conservation of environment, clean water and air 

and others " ,T 

• Degree of harmony, law and order in the community 
Initiatives taken for spiritual fulfillment 

■* Based on ATLP-CE. . > 
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2.9 Convergence of Indicators with Monitoring Tools 

After developing indicators, the next immediate task for planning a monitoring system is 
the development of tools. It is important to know how to transform the indicators into 
various monitoring tools. There can be a variety of monitoring tools such as survey 
forms, questionnaires, progress reports, visit checklists, checklists for organizing focus 
group discussions, and others. It is neither necessary nor feasible to integrate all 
indicators into a single monitoring instrument. Again, all information would not be 
required and used simultaneously. As mentioned earlier, programme staff needs for 
information at various levels are different. It is therefore useful to systematically 
develop the monitoring tools carefully, taking into account these needs and the required 
frequency of information gathering. In developing tools, the appropriate mode of 
collecting information should also be kept in mind. This section attempts to explain how 
monitoring tools can be developed based on the indicators identified for monitoring 
purposes. However, these steps are not prescriptive or exhaustive. The same 
information may be generated through the use of other methods and tools. 

2.10 Suggested Steps for Developing Tools Based on Monitoring 
Indicators 

Step One: Select an indicator specifying the type of information needed. For example, 
information about the percentage of learner drop-out may require other data 
on gender or income level. 

Step Two: Identify the source of information, i.e., who can best supply the information. 

In the case of drop-out data, for example, it is the teacher who can best 
provide all of the details. 

Step 77»ree:Work out how the information may be collected accurately. There can be a 
number of ways - through a written report or a physical visit, or from group 
discussion. For example, information about visits of extension agency 
personnel to the literacy centres can be collected from the reports of teachers 
or through discussions with learners. 

Step Four: Check what types of tools would be appropriate to collect, to record and, if 
necessary, to transmit information. It might be a report form, a checklist, or 
a diary kept by the teacher or supervisor. 

Step Five: After the initial exercise of linking the indicators with the mode of collecting 
information, the next essential step is to consolidate the information 
collection system. This is very important in order to save time in collecting 
information. If the teacher, for example, is asked to provide information at 
a given time, it is better to ask for such information in a package. However, 
it largely depends on whether the frequency (monthly, quarterly, etc.) and 
the mode of collection (discussion, report, etc.) of the information are 
the same. Simplify the integration of information to make it easy to 
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communicate and user-friendly. Complicated instruments may elicit 
misleading information. 

Step Six: The next important stage is to decide who else in the project management 

system (at village or district level) requires similar information and how 
often. It is also .advisable to determine whether the same monitoring 
instrument would meet the requirement or a new set of tools needs to be 
developed. The same sequence of steps for developing tools may also be 
applied at this level. 

The following table gives examples of how to apply the steps earlier mentioned, showing 
how indicators lead to the development of monitoring tools. 

Table 2.3 Sample Illustrations on the Development of 





Monitoring Tools based on Indicators 


Monitoring indicators 
(at learning centre level) 


Possible 
source of 
information 


Mode of 
collection 


Frequency of 
information 
required 


Monitoring 

tool(s) 


: 1 . Percentage of targeted 
j learners enrolled 


Facilitator 


j Review of enrollment 
| records 


One time at the 
beginning of the 
project 


i Report form 


| 1 2. Percentage of women/ 
| disadvantaged people 


Facilitator 


j Review of enrollment 
i records 


One time at the 
beginning of the 
project 


: Report form 


; 3. Percentage of average 
j attendance by gender/ 

: income 


• Facilitator 


: Review of attendance 
i records 


Monthly/ 

quarterly 


j Report form 


j 4. Percentage of achieve- 
: ment of learners accord- 

: ing to literacy levels and 

\ functional skills 


; Facilitator/ 
; supervisor/ 
j learners 


j Physical test of 
j achievement 


Monthly/ 

quarterly 


j Test reports 


j 5. Percentage of learner use 
j of literacy skills with 

j kinds of skills and 

: frequency of use 


! Facilitator/ 

| supervisor/ i 

j learners 


j Focus Group Discussion 
1 (FGD) 


Quarterly 


j Checklist for 
j FGD 


; 6. Percentage of learners who 
i drop out and reasons (by 

i gender/income) 


; Facilitator/ j 

! supervisor 


i Review of attendance/ 
; enrollment records 


Monthly/ 

quarterly 


j Report form 


; 7. Percentage of learners with 
i access to education materials 


i Supervisor/ 
| learners 


j Physical visit to centre 


Monthly 


| Supervisor's 
j visit checklist 


i 8. Percentage of attendance of i 

; : committee members at 

| meetings 


Facilitator/ i 

supervisor/ ; 

j community i 

members 1 


| Physical presence in 
; meeting/ meeting records 


Quarterly 


! Report form/ 

; supervisor's 
; visit checklist 


j 9. Percentage of centres visited j 
; by community leaders/ 

: extension agency personnel i 


Facilitator/ i 

supervisor/ j 

i learners j 


i Visit records 


Quarterly 


: Report form/ 

! supervisor's 
1 visit checklist 


j 10. Percentage of learners 
: satisfied with the contents of i 

: the curriculum \ 


Facilitator/ j 

supervisor/ 

learners 


Focus Group Discussion 
(FGD) 


Quarterly 


! Checklist for 
FGD 


ill. Percentage of family (of j 

: learners) where changes j 

j occur as result of literacy and | 

i | types of changes * i 


Facilitator/ j 

supervisor/ : 

learners j 


Focus Group Discussion 
(FGD) 


Quarterly 


! Checklist for 
| FGD 


: 12. Percentage of graduates j 

| ; linked with post-literacy j 

programmes j 


Facilitator/ : 

supervisor/ j 

learners \ 


Review of enrollment 
records 


One time 


Report form 
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Chapter Three 

DESIGNING METHODOLOGIES IN MONITORING 



This chapter aims to enable programme planners and jj 

practitioners of literacy and continuing education 

programmes to: 

* gain deeper understanding of monitoring tools / jj 
instalments and methodologies; 

* identify various levels of programme implementation j 
where monitoring data may be collected; 

* identify the frequency of monitoring activities; 

* understand the methods of data analysis; 

* estimate monitoring costs; and 

* develop a training programme on monitoring for mid- j 
level practitioners of literacy and continuing education jj 
projects . 
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3.1 Monitoring Tool and Instrument Defined 

A monitoring tool is an assessment device that aims to determine the progress of an 
ongoing programme. It is also called a monitoring instrument. A monitoring tool should 
be able to identify gaps or weaknesses to enable decision-makers to revise or change 
programme activities and to undertake appropriate corrective action. For each indicator, a 
measurement tool needs to be developed. The quality of monitoring depends on the 
efficiency with which the tools are prepared and used. 

3.2 Types of Monitoring Tools/Instruments 

In the process of selecting monitoring tools, the programme monitors should be familiar 
with the different types of tools that are most suitable for collecting data/information for 
various purposes and at different levels of programme structure. Some of these 
tools/instruments are: 

♦ Questionnaire 

♦ Checklist 

♦ Programme documents 

♦ Observation schedules 

♦ Interview guide 

♦ Diary/joumal 

Questionnaire. A questionnaire is a list of structured or unstructured questions to be 
answered by a number of people. These questions should be simple, clear, and easy to 
understand. The language and terminology used should be engaging and unambiguous. 
Some of the questions may be phrased in such a way as to allow for open-ended 
comments. The instrument should be prefaced by brief instructions and a statement of the 
purpose of the questionnaire. 

In developing a questionnaire, one should try to avoid unnecessary questions. However, 
standard demographic information such as age, gender, occupation, marital status and 
education level will help in interpreting the overall responses. Questions about sensitive 
issues related to the community may also be asked in order to determine their relationship 
to programme implementation [see Appendix I]. 

Checklist. A checklist is a prepared list of items that are to be observed at the project site. 
It consists of the monitoring indicators earlier identified and developed in order to 
determine the progress of the programme. It should be clear and simple. Photographs and 
illustrations can be included in order to capture project profiles more accurately. The 
learners are welcome to fill in their own thoughts, insights and perceptions. It can be in the 
form of a task checklist, quantitative data sheet or qualitative data sheet. Examples are in 
Appendix I. 

Programme Documents. Records and documents are important sources of monitoring 
data. Analysis involves a random selection of content and statistical techniques for making 
general assertions. The analysis of these records and documents may be quantitative or 
qualitative. Some of them may already be available in the community and the learning 
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centre, for example, the village profile and the attendance record book. This information 
could be helpful fo^all levels in planning, implementing and monitoring ongoing or new 
projects. 

Observation Schedule . During field observation, data are collected on a random basis. 
Observations must be recorded, a task made simpler by setting structured objectives before 
going to the field. Available records at the site could be used as inputs to the observation 
record. Some examples are checklists, maps, charts, diaries and schedules. Caution should 
be taken when using this method. It should be checked by other methods such as 
interviews, group discussions and hard evidence. For example, books that look new may 
be a sign that they have not been touched by programme beneficiaries or used by the 
instructor. 

Interview guide . Interviews can be structured or unstructured in nature. In an unstructured 
interview, the respondent may describe topics in his/her own words, make suggestions and 
contributions. The interviewer encourages, facilitates and provides reassurance and 
reinforcement. Because it is carried out in a face-to-face situation, it gives the interviewer 
an opportunity to probe further and ask questions that will validate earlier responses. 
However, he or she should consider several factors before, during and after the interview, 
most notably: 

♦ The culture and norms of the respondent; 

♦ Convenience of the respondent; 

♦ Explanation of the purpose, methodology and confidentiality of the answers; 

♦ A good rapport and trust in regard to the information gathered; and 

An example of an interview guide is shown in Appendix I. 

Diary/journal A diary or journal is a record of events and insights made by either 
the programme administrator at the field level or a programme beneficiary or 
group of beneficiaries. It may be a continuing account or a periodically prepared 
one. It is a useful monitoring tool because it is an eyewitness account of 
programme activities. 

3.3 Characteristics of Good Monitoring Tools 

Monitoring tools should have the following characteristics: 

♦ User friendly 

♦ Improvement-oriented 

♦ Interventionist • ' 

♦ Self-instructing 

♦ Comprehensive 

♦ Independent of evaluator bias . 

♦ Simple language/understandable 

♦ Time-saving/economical 

♦ Easy to analyze 
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3.4 Development of Monitoring Tools 

The quality of a tool depends on the efficiency with which it is prepardil and used. Some of 
the important things to consider in developing a monitoring tool are: 

A list of items for measurement. Prepare a list of items to be measured as reflected in the 
statement of specific operational objectives, structured curriculum and the components of 
the programme. Separate them into items for observation, interview and/or testing. 

Examples of such items are: 

♦ Absentees during the rainy season; 

♦ Relevance of topics; and 

♦ Number of participants by the end of the year. 

Structuring the items. Questions are stated in short, simple terms without ambiguity. A 
measurement scale, index or rating scale for the items is constructed so that items are truly 
representative and measurable The weight or relative importance of the different items is 
specified by a single measurement value Scores are calculated on the basis of a 
framework of weights. 

Validity and reliability. The validity and reliability of the data/information depends on 
several factors: 

♦ The variables that influence each other; 

♦ Attitude/mood of participants during the data collection; 

♦ Social position of the participants; 

♦ Lack of training on the part of the facilitator; and 

♦ Group interviewing. 

To ensure the validity and reliability of monitoring, monitors must make sure that the 
following conditions are in effect: 

♦ The instrument is administered in the same time frame, in the same manner, to 
different participants; 

♦ The participants are aware of the purpose and procedures of monitoring; and 

♦ All sources of data/information are recorded and kept confidential. 

Pretesting of tools. Pretesting is carried out to determine the level of performance 
of the monitoring tool before administering it to the intended respondents or 
beneficiaries of a programme or project. Pretesting helps ensure accuracy and 
validity of the outcome and also helps to identify possible problems prior to the 
testing proper. The tools should convey the same meaning to all respondents, 
clearly and without any ambiguity. The tools can be revised again and again as 
necessary because this process will help bring about some revisions and 
modifications to the monitoring approach as well as to the tool itself. 

Selection of tools. Proper selection of monitoring tools will help ensure the accuracy of the 
information collected. Here are some of the factors to consider when selecting a tool: 
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♦ Profile of respondent; 

♦ Level of respondent; 

♦ Level of monitor/evaluator; 

♦ Purpose; 

♦ Size of sample; 

♦ Time; 

♦ Resources; and 

♦ Financial considerations. 



Figure 3.1 Examples of Tools Commonly Used at Different Levels 




3.5 Major Sources of Monitoring Data 

There are three major sources of monitoring data: 

♦ Day-to-day records of the operation of an education system or literacy programme 
provide substantial data on personnel, target groups and various activities. Some 
examples include attendance data on learners, or expenditure data on personnel, 
materials and equipment. Learner attendance registers, instructors’ and supervisors’ 
diaries and data sheets/cards are also good sources of monitoring data. 

♦ In-house periodic monitoring conducted by an organization may serve as a good 
source of monitoring data. Regular agency monitoring tasks are a review of the 
organizational structure, functions and activities. Most education systems and 
institutions conduct periodic monitoring on progress of pupil performance, physical 
facilities, environmental conditions, availability of educational materials and other 
information deemed necessary for efficient management of the institution. 

Similarly, at the literacy centre level, periodic tests are conducted to find out the 
achievement levels of learners. Periodic monitoring may be conducted in structured or 
unstructured form. Learner evaluation may be conducted either by administering the 
necessary tests or while playing some educational games. As these games and 
activities will be used as monitoring tools, they should be developed carefully, keeping 
in mind the monitoring indicators. Further, they should be administered carefully at 
the appropriate time. 
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♦ Special data collection related to issues of policy or strategies may occur whenever 
there is inadequate information on these issues. Such special data collection activities 
are often based on samples and have the advantage of immediate relevance, greater 
detail on given topics and a reduced time lag between the need for information and its 
generation and, ultimately, its use by programme planners and managers. 

3.6 Methods of Data Collection 

There are various methods of data collection. The programme monitor decides on the 
methodology of data collection when he or she selects the monitoring tools. Some 
commonly used data collection methods are: 

♦ Interviews; 

♦ Observations; 

♦ Focus group discussions; 

♦ Documentation and case studies; 

♦ Surveys; and 

♦ Analysis of records and data. 

In order to choose the most appropriate method of data collection, it is helpful to ask the 
following questions: 

♦ Can this method generate the needed information? 

♦ Will this method be acceptable to the people who will be involved? 

♦ Does it fit in the identified time frame for monitoring the particular activity or 
programme component? 

♦ Is there any equipment or back-up support needed and, if so, is it available? 

Whichever method is selected, the monitor should be cautious. The dangers of being 
overwhelmed by data have already been mentioned earlier. Monitors should collect only a 
moderate amount of information; they should seek more data only to validate what they 
have or if the existing data are confusing, contradictory or inconclusive. 

Efficient monitoring can be done by: 

♦ Devising economical methods of gathering information, e g., by using existing data 
and/or reviewing and analyzing the answers provided earlier by respondents; 

♦ Collecting data in different ways by using different instruments and methods such as a 
questionnaire, written assignments, discussion, etc.; 

♦ Collecting the data from different samples, e.g., collecting data from one source and, 
for review purposes, gathering data from other sources; 

♦ Using participatory methods to collect data by involving the respondents in the data 
collection process; and 

♦ Planning the monitoring activities systematically. 
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3.7 Some Guidelines for Data Collection 

♦ Status differences or difficulties in relationships make some people unwilling to be 
open about their ideas or opinions. Establish a fruitful and collaborative relationship 
with them. Inform them about what you are doing and why you are doing it. 

♦ If you need assistance from colleagues or other field workers, provide them 
appropriate training and orientation 

♦ Even though you cannot foresee natural calamities, be aware of the daily and seasonal 
patterns of respondent activities so that you can plan your visit accordingly. 

♦ Anticipate the problems you might face in the course of data collection and try to 
overcome them. 

♦ If possible, use a tape recorder to record your interviews. 

♦ Collect only the amount of data that is manageable yet sufficient to provide needed 
information. 

♦ Make sure that individuals do not dominate a group discussion and push it in certain 
directions. 

♦ Be aware that some people may hesitate to express criticism that may hurt others. 

3.8 Collection of Data at Various Levels 

Data collected at various levels in the form of periodic returns give valuable information. 
They may be collected at the following levels: 

Learning centre level The instructors' initial reports, monthly and final reports normally 
provide information on the attendance record, drop-out rate, problems faced, achievements 
of learners and success stories. In many post-literacy and continuing education centres, 
instructors' records give information about learner interest in certain learning materials, 
their need for various skill development activities and other additional information 

In addition to instructors' records, the learners' exercise books are also sources of 
monitoring data. Detailed information about their progress, time taken in achieving 
various levels and problems in learning may be obtained from these books. 

At the learning centre level, the instructor should be motivated to fill in the monitoring 
sheets and attendance register regularly. These reports should be collected by the 
supervisor or monitor. With the information received, programme activities at the field 
level can be strengthened and corrective measures taken to address weaknesses. 

Village level. At the village level, data may be collected from the village co-ordinators' 
records. These records may consist of: 

♦ The village co-ordinator’s diary containing observations and notes during visits to the 
literacy/CE centres and also comments received from the instructors; 

♦ The village co-ordinator’s periodic reports to the district office. This report is mainly 
based on information received from the various literacy/continuing education centres in 
the village; and 
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♦ The village co-ordinator’s registers and existing records. 

In addition to village co-ordinators, the community members themselves are good sources 
of data. They provide information about various activities organized in the field and their 
impact on the target community as a whole. 

The village co-ordinator or monitor at the village level should: 

♦ Review and study the areas of monitoring as well as the indicators and criteria for 
monitoring; 

♦ Set up a time schedule for conducting the monitoring activities in each place; 

♦ Estimate the expenditure for transportation and other expenses for conducting the 
monitoring activities; 

♦ Select the monitoring tools or instruments for collecting the information about 
instructor or facilitator performance and other factors that lead to the success or 
failure of the programme; 

♦ Collect all the related data for analysis; and 

♦ Report and make plans for improvement. 

District level. The reports prepared by village co-ordinators are consolidated and analyzed 
at the district level before transmission to the appropriate authorities. District-level reports 
usually consist of periodic summaries of district education activities and district data 
registers. 

In addition to this information, data may be collected from people's representatives, NGOs, 
district administration offices, various development departments, and other appropriate 
agencies. 

Provincial/state level. The reports from the provincial or state office to the national head 
office may indicate the following, number of classes, number of learners, drop-out 
percentages, achievement level of learners, financial position and the strength and 
weakness of administrative structures in the implementation of the literacy/continuing 
education programmes. Usually, the information is based on the reports received from 
various districts. Provincial/state data registers are also potential sources of monitoring 
data. 

At the provincial level, monitoring may be done on a quarterly basis and the data 
processed and sent to the state Department of Non-formal Education or other similar 
agencies. 

National level. National-level data are based on the reports received from regional sites, 
provinces or states. Therefore these reports should be based on an objective and balanced 
interpretation of the performance of the implementing agencies in order to identify the 
strong points of the programmes. They should also suggest concrete steps for overcoming 
weaknesses or difficulties encountered during the implementation of activities. 

The national headquarters maintains a national data register and reports annually to higher 
levels in the Government about the programme progress and prepares reports for the 
consumption of other publics such as the media. 
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The department which is responsible for the literacy and continuing education programmes 
has to determine a set of goals for the programmes according to the policies of the country. 
This department should set up a responsible division or identify a programme manager 
who will be responsible for monitoring activities. The functions of monitoring at this level 
are usually as follows: • 

♦ To establish the monitoring functions of each level; 

♦ To coordinate and cooperate with related networks in undertaking monitoring 
activities; 

♦ To establish the policy, goal and objectives of the programmes; 

♦ To determine the area, criteria and indicators for programme achievement; 

♦ To design a management information system for data collection at each level of the 
country; 

♦ To design the tools or instruments for data collection in each phase of the ongoing 
programme; 

♦ To analyze the needed data according to the time frame of each unit of the country; 

♦ To take corrective measures in time. This particular step should be properly planned 
and executed in the overall scheme of monitoring literacy and continuing education 
programmes; and 

♦ To prepare reports and give suggestions for further improvement to policy makers. 

3.9 Flow of Information 

The flow of monitoring information should be both up and down: 

Bottom up, Included in this flow of information are data collected through periodic 
reports, visits, meetings between functionaries of different levels, training and evaluation 
sessions, documentation, and the collection of success stories. 

Top down. Instructions, circulars, training programmes, interagency meetings and 
seminars usually direct information from the national level to the levels of the district, 
village or learning centre. 

The following illustration shows the flow of information in the monitoring and evaluation 
of literacy and continuing education. 
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Figure 3.2 Flow of Information in Monitoring and Evaluating 
Literacy and Continuing Education: An Example 



FROM STATE TO STATE 

AND NATIONAL LEVEL AND NATIONAL LEVEL 




Source: National Literacy Mission. Government of India, New Delhi. 1988. 



3.10 Time Frame/Frequency of Data Collection 

When designing the monitoring plan at each level, the programme monitor should prepare 
the time frame for conducting the monitoring activities at different phases, namely after 1 
month, 3 months, 6 months, 1 year, etc. Additionally, he or she should provide for 
flexibility in the frequency of monitoring activities. For example, the supervisor may visit 
the literacy class once a month while "the instructor tests the literacy level of learners once 
every 3 months. The frequency of monitoring at each level depends on the objectives of 
the programme, but sometimes the frequency of occurrence may change if there are 
unexpected problems or difficulties that have to be overcome directly. The programme 
director or manager has to make the decision to revise schedules. 

3.11 Analysis of Data 

Data are useful only when they are appropriately analyzed. In general, the analysis of 
monitoring data addresses the following three issues: description of the programme; 
comparison between sites; and conformity of programme implementation to its design. 
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Description of the programme. Determining the extent to which a programme has 
implemented its planned activities largely depends on a full understanding of the 
programme itself Therefore the monitor should have access to the description of the 
programme and should be familiar with the following: 

♦ Programme goals, strategies, coverage and reach; 

♦ Types of services delivered, the intensity of services given to participants/clients; 

♦ The reactions of participants to the services delivered; and 

♦ Achievement/progress of the programme based on testimonials and other forms of 
feedback. 

Descriptive statements may take the form of narrative accounts, especially when 
monitoring data are derived from more qualitative sources. Quantitative analysis is also 
useful, and sophisticated analytic methods and measures are increasingly being developed 
to accurately describe programme progress. 

Comparison between sites . When a programme includes more than one site, the 

programme monitor should be concerned with the differences in implementation between 
the sites. Comparison of performance among various sites provides an opportunity for 
understanding the sources of diversity in project implementation and outcomes. Some 
points for comparison include differences in staff, administration, targets or surrounding 
environments. Comparison can facilitate efforts to achieve standardization. 

Conformity of programme implementation to its design . The degree to which the 
ongoing operations of a programme match its original intent is a major concern of 
monitoring. Discrepancies between the two may lead to efforts to move the implementation 
of a programme closer to the original design or to a re-specification of the design itself. 
Such analysis also provides an opportunity to gauge the appropriateness of impact 
evaluation and, if necessary, to plan a more appropriate formative evaluation (see Section 
5.4) in order to develop the desired convergence of design and implementation. 

3,12 Procedures for Analyzing Data 

Once the data have been collected, they have to be turned into usable information. This 
requires a process of analysis: sorting and collating the data and reducing them to 
manageable proportions. This must be done with care so as not to distort or bias the 
results. Analysis can be carried out by an individual or by a group of monitoring staff. 

If the information has been collected in the form of tick-box responses, the task is one of 
counting. If the material consists of responses to open-ended queries, you should read 
everything with care. The important task to be done is to thoroughly review the responses 
to the monitoring instrument. 

Then the group begins to draw up a set of categories that covers the issues earlier 
identified and sees if they can be applied to the questionnaire or observation data. As the 
work continues, the group may find the need to add new categories or perhaps reduce the 
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original list. They can keep modifying until they have a workable set. This process is 
seldom as difficult as it may sound. Categories and patterns emerge quite quickly. 

Once the categories are clear, the group goes through the information again to note the 
span and frequency within each category. 

For instance, learners use the term “boring” to describe the following perceptions: 



too difficult 


### 


12 


too easy 


/// 


3 


covered in another subject 


llll 


4 


not relevant 


II 


' 2 


not interesting 


m 


6 



The group may want to do something more complicated. For example, they may want to 
check if there is a connection between an instructor’s responses to a particular question 
and the numbers of years he or she has been teaching. If there is a connection, it probably 
will emerge quite quickly. 

Another approach is to gather together a group of perhaps six to ten participants to 
analyze a large number of items; each person is given a task, as follows: 

♦ If there are tick boxes to tally, each person records the scores for the first question. If 
there are more open-ended questions, each individual reads the answers to the first 
question, and writes down a brief summary of the responses. It is possible to add one 
or two representative quotations to the summary in order to add some “colour” to the 
final report. 

♦ After this activity, each person then reads his or her summary to the whole group. It is 
advisable to identify apparent inconsistencies A question to ask in this exercise is. 
Are some responses being biased by those who make the summaries, or are they being 
synthesized logically? 

♦ This process may be repeated until all the questions have been analyzed in this way. 

♦ Each person then collects the summaries and produces a final summary, including 
quotations, if desired. These are also read aloud to provide a final opportunity to 
check the balance of reporting. 

♦ These summaries can be put together to serve as a basis for the monitoring report, 
with perhaps the inclusion of an agreed-upon introduction and conclusion. 

It is advisable to conclude by discussing lessons learned and insights gained, as well as 
what has been learned about the monitoring activity itself - about the methods used, about 
constructing a questionnaire, about carrying out and recording interviews or observations, 
and the total monitoring process. 
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3.13 Statistical Methods for Data Analysis 

After the data have been collected, various methods may be used for data analysis. 

Some examples are: 

♦ Percentages may be used to determine rates, such as the enrollment rate, drop-out rate 
and the attendance rate. 

♦ The following formula may be used for measuring the effectiveness of coverage: 



Coverage efficiency = 100 x Number in need served 

Total number in need 



Number not in need served 
Total number served 



♦ The ratio scale may be used for determining ratios and proportions. 

A 3 to 5-point scale (Likert scale) measures the satisfaction or attitude of clients. 
Percentage, frequency allocation and standard deviation are normally used to analyze 
these data. 

Content analysis is normally used for qualitative data. 

Existing software programmes for coding, sorting, computing and analysis, such as 
the SPSS, may also be used. 



3.14 Training 

The ATLP and ATLP-CE identify three levels of personnel involved in literacy and 
continuing education. Level A personnel under ATLP are those individuals who make 
policy, and plan and implement literacy training programmes in their country. Level B 
personnel consist of a cadre of professional supervisors and trainers who operate in key 
locations throughout a country and Level C staff are grassroots personnel working directly 
with learners in the community. 

This manual focuses on the training of Level B literacy personnel They are usually 
provincial district supervisors and trainers of staff who conduct actual literacy/continuing 
education classes. In literacy programmes, they are fully involved in literacy work, but in 
continuing education programmes there are generally two categories of Level B personnel. 
The first category remains entirely involved in literacy programmes while the second may 
have dual responsibility in literacy as well as in different areas of continuing education. 
All these personnel have to be given orientation on the importance of monitoring and 
evaluation and should be provided various skills and competencies for monitoring their 
programmes. 

Level B personnel have to monitor the Level C workers. They have to be trained in 
monitoring skills which are critical to their work. In the case of continuing education, 
monitoring skills are very different for each of the six types of continuing education 
programmes (see Section 2.8). In some cases, like post-literacy programmes, the 
monitoring would be direct and continuous, while in other programmes it would occur at 
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certain intervals. Level B personnel should also be able to help Level C personnel in 
monitoring and supervising their own staff who perform actual monitoring activities: 

For programmes such as basic literacy training in which there are clearly defined stages of 
achievement, monitoring techniques are simple and easy to implement. On the other hand, 
the outcomes of continuing education are more varied and wide ranging, registering 
various levels of achievement; some are even quite specialized and technical in nature. 
Therefore, Level B personnel involved in monitoring continuing education programmes 
require intensive training in programme monitoring methods and strategies. 

In general, all personnel who are involved in monitoring literacy and continuing education 
programmes should have specific knowledge and skills to monitor effectively, including: 

♦ A clear understanding of both monitoring and evaluation; 

♦ Basic techniques of monitoring, including indicators and tools; 

♦ Specific monitoring and evaluation techniques to assess skills of leamers/neo-literates 
according to competency level using a variety of techniques; 

♦ Techniques to monitor the various activities of literacy and continuing education 
programmes; 

♦ Techniques of data processing and analysis; and 

♦ Report writing and dissemination of information. 

Training programmes for monitoring personnel should include workshops on metho- 
dologies of monitoring, the design of various tools and instruments, the organization and 
analysis of data collected from the field, interpretation of data, and formulating 
recommendations. 

The monitoring methods used should not be threatening and should be sensitive to the 
socio-cultural contexts of the target group. Equally important, the persons involved in 
monitoring should enjoy the work. 

The following is a sample training schedule for Level B personnel in monitoring literacy 
and continuing education programmes: 
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Days 


Topics 


Methodology 


i Day I 


- Registration 

- Introduction of participants 

- Introduction to the training programme 


Breaking the ice 


| 


- Expectations of the participants 


Presentations/ discussions 


| 


- Sharing the experiences on 


Lecture supported with 


| 


monitoring of literacy/CE programmes 


transparences (OHP) 


jf 


- Monitoring - definitions, context. 






objectives and functions 
- Monitoring in the context of literacy/ 
CE programmes 


Group discussion 
Familiarization with manual 


1 Day II 


- Characteristics of good monitoring 


Reading and group work . 1 




system 

- Tasks of programme monitor 


Presentations and group work 




- Monitoring plans and major steps in 






preparing monitoring plan 


Discussions and presentation 


j Day lH 


- Monitoring indicators 

* concepts 

* features of good indicators 

* types of indicators 

* development of indicators 

- Convergence of indicators with monitoring 
tools 


(whole day) 

Presentation and preparation of 


1 Day IV 


- Monitoring Tools 


tools 




* types of tools 


Group work 




* development of tools 






* pre-testing of tools 

* selection of tools 






- Data collection mechanisms in literacy/CE 

* methods of data collection 

* data collection forms 

* collection of data at various levels and frequency 
of data collection 

* roles of various functions in data collection 


Demonstration 

Exercise 

Group work | 


j Day V 


- Data analysis 

- Report writing and dissemination 

- Preparation for training of Level C workers 

- Validating session 


Demonstration and practice; 
Writing exercise; ! 

Development of training 
schedule 



3.15 Cost of Monitoring 

For monitoring purposes, the data collection 
and continuing education programmes at each 
integral part of any development programme, 
be included in the total project cost. 



mechanism should be built into the literacy 
level. Participatory monitoring should be an 
Therefore the cost of monitoring should also 
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Usually the cost of monitoring is calculated based on the following assumptions and 
factors: 

♦ Number of villages to be monitored; 

♦ Number of clusters of villages to be monitored; 

♦ Number of persons involved in the monitoring; 

♦ Number of monitoring co-ordinators; and 

♦ Number of agencies/organizations to be visited and records to be checked and 
collected, as well as meetings to be organized. 

Based on the above data, the cost for regular monitoring is calculated as follows: 

♦ Salary of the co-ordinator; 

♦ Salaries of the monitors; 

♦ Preparation and production of tools; 

♦ Travel and transportation costs; 

♦ Cost of organizing meetings in the field; 

♦ Training of monitoring personnel; 

♦ Stationery and other materials; 

♦ Computer rental or purchase; 

♦ Salaries of computer operators; and 

♦ Miscellaneous. 



Chapter Four 



REPORTING AND DISSEMINATION 
OF MONITORING RESULTS 



This chapter aims to enable programme planners and 
practitioners of literacy and continuing education 
programmes to: 

* gain deeper understanding of reporting and dissemi- 
nating monitoring results; 

* understand the concept of Educational Management 
Information System (EMIS); 

* realize the importance and need for establishing an 
EMIS; 



o 
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4.1 Reporting on the Progress of Programme Implementation 

Programme managers should keep in mind that the ultimate objective of monitoring is to 
communicate relevant findings to the decision makers, practitioners, community members 
and other stakeholders in the programme. Thus it is essential to develop a system of 
reporting that will efficiently and speedily disseminate the monitoring results and 
recommendations to all concerned parties and groups. 

A sample reporting system is illustrated as follows*: 



( 4 ) 




*Source: Dossier of Evaluation Instruments for Literacy Programmes , edited by Josef Miller and 

Anja Dietrich, Centre for Education Science and Documentation, Bonn, Federal 
Republic of Germany, 1989, p. 79. 

The chain of reporting starts from the instructor who reports to the supervisor. The 
supervisor in turn reports to the project officer. The latter is expected to submit reports to 
the national centre for literacy and adult education and to the regional community 
development office. 

The reporting system is developed in such a manner that it has linkages back to the 
planning of the programme and forward to the control and evaluation stage of the 
programme. 

There are some key points on reporting procedures which may lead to better use of 
monitoring findings. 

♦ Reports should be made with reference to the framework, definitions, indicators and 
hypotheses established in advance. This would help to avoid ambiguities and 
disagreements concerning what the programme is intended to accomplish. 

♦ Findings should be reported promptly to allow quick feedback of information for 
changes that may need to be made. 

♦ Documentation of the programme components should be emphasized. Such detailed 
information on the programme process will clearly point out corrective measures that 
are needed and guide decision-makers in selecting among alternatives for future action. 

In general, the reporting system may include the following information from the field: 
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♦ Literacy programme class profiles; 

♦ Instructor profiles; 

♦ Monthly, report I by the supervisor to the project officer; 

♦ Monthly report II by the supervisor to the project officer; 

♦ Monthly consolidated report by the project officer to the regional or provincial centre 
and the national centre for literacy education; 

♦ Monthly report by a partner agency or NGO to the national centre for literacy 

education; * ' ? 

♦ Monthly consolidated report on the literacy programme by the national centre for 
literacy education for information and feedback; and 

♦ Periodic and yearly report by the national centre for literacy education for information, 
feedback and documentation. 

At different programme levels and at various stages of programme implementation, the 
reporting forms may be used by monitoring personnel as well as shared with partner 
agencies or NGOs involved in the project. Sample forms for reporting monitoring 
information are provided in Appendix II. 

4.2 Establishing an Educational Management Information System 
(EMIS) 

EMIS defined. An Educational Management Information System (EMIS) is an organized 
structure that governs the collection, processing, analysis, reporting and use of 
information. It is a collection of component parts that include inputs, processes, outputs 
and feedbacks that are integrated to achieve a specific objective. It is a system for 
managing a large body of data and information that can be readily retrieved, processed, 
analyzed, and made available for use and dissemination. 

The need for establishing an EMIS. The main purpose of an EMIS is to integrate 
information about related activities and make the information available in a comprehensive 
way to a variety of users. Every country has some arrangements for reporting about 
literacy and continuing education programmes. Thus there is a need to strengthen this 
aspect in a systematic manner to facilitate wider and more effective dissemination of 
information gained through programme implementation and the exchange of experiences 
and expertise in literacy and CE programmes. 

The main justifications for giving priority to an EMIS are: 

♦ The rapid expansion in the physical size and geographic coverage of the education 
system; 

♦ The increased complexity of educational activities in terms of their number and variety 
of outcomes; 

♦ Need for more efficient use of resources; 

♦ Greater accountability requirements; and 



44 O Practitioners' Manual on Monitoring and Evaluation 



♦ The improved availability of relatively low-cost technologies for dealing with large 
data sets. 

The importance of an EMIS system and procedures within educational management can be 
expected to substantially grow into the next century as concerns about the financing and 
management of education escalate in both developed and developing countries. 

The operation of a successful EMIS involves five major steps: 

♦ Needs identification; 

♦ Data collection; 

♦ Data processing and analysis; 

♦ Information provision (reporting); and 

♦ Use of information. 

EMIS structure. In reality, the people of the community are the main beneficiaries of an 
EMIS. The basic source of information about literacy and continuing education are the 
learners themselves. Information originating at their level has to flow through various 
institutional and individual channels of the system, providing necessary feedback at the 
appropriate time. An example of such a structure is given here to illustrate the channels of 
information flow as well as feedback. 



Channels of Information Flow and Feedback at Different Levels 




Level 5 



Level 4 



Level 3 



Level 2 



Level 1 



Source: 



Adapted from Environmental Education Handbook for Educational Planners. National 
Institute of Educational Planning and Administration. New Delhi. 1990. 
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4.3 Information Flow in the EMIS 

The flow of information is crucial in an EMIS network. Within a well-established EMIS, it 
is imperative that both basic data and feedback information flow both ways between the 
EMIS and other units. Information products should be provided to as many other 
information outlets as possible in order to reach the maximum number of users. 

Figure 4.1 illustrates the flow of information from the EMIS to the users at different 
levels. 



Figure 4.1 Information Flow 



EMIS DATA 




LEVELS DECISION-MAKING/ 




INFORMATION 


AND INFORMATION 




DATA UTILIZATION 




PRODUCTS 



STUDENT 

MIS 



CURRICULUM/ 

PROGRAMME 

MIS 



PERSONNEL 

MIS 



FINANCIAL 

RESOURCE 

MIS 



PHYSICAL PLANT 
AND FACILITIES 
MIS 



•INSTITUTIONAL 

PROFILE 

MIS 



L 



OTHER 

MIS 
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(GO AND NGO) 



I 



REGIONAL/ 
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STATE 



I 
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I 
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PROJECT 
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Source: UNESCO, PROAP, ( 1 998). Educational Management Information System , Bangkok, p.9. 
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4.4 Some Additional Guidelines for an EMIS 

♦ A simple but robust EMIS can help in drawing out conclusions about problems and 
progress. 

♦ You should select a few indicators and collect information systematically on them at 
all levels - from the community to the national level. 

♦ Information should be contextual. 

♦ A local and decentralized EMIS is most useful at the field level. 

♦ Gender perspectives can be incorporated into all aspects of an EMIS. 

♦ It is useful to assess the processes by which the local community facilitates 
participation and achievements. 

♦ An efficient EMIS helps in the assessment of teacher competence, training, motivation 
and teaching environment. 

♦ It also helps in profiling learning competencies and existing levels of achievement. 

♦ EMIS records should help in determining the need for improved teaching/leaming 
materials. 

♦ A good EMIS should tell us something about the participation of girls and women, as 
well as those belonging to disadvantaged groups. 

♦ An efficient EMIS should generate information on the participation of minorities. 

♦ EMIS records also help in providing information about physical facilities. 



PART I I: EVALUATION 
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Chapter Five 



UNDERSTANDING EVALUATION CONCEPTS AND 
THE PROCESS OF DEVELOPING AN EVALUATION PLAN 



j This chapter aims to familiarize practitioners of !j 
| literacy and continuing education programmes with j 
| concepts of evaluation and the process of developing j 
| an evaluation plan. Specifically, they will be able to j 
j understand: 

* definitions of evaluation; j 

j! * the need for evaluation; jj 

| * the scope, coverage and functions of evaluation; 

| * categories and types of evaluation at various stages of 

|| programme implementation; and 

j * the process of developing an evaluation plan. jj 
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5.1 Evaluation Defined 

The following are some definitions of programme evaluation: 

♦ Evaluation is the process of delineating, obtaining and providing useful information for 
judging decision alternatives (Stufflebeam, 1971), 

♦ Evaluation is the determination of the worth of a thing. It includes obtaining 
information for use in judging the worth of a programme, product, procedure, 
objective or the potential use of alternative approaches designed to achieve specific 
objectives (Worthen and Sanders, 1973). 

♦ Evaluation is the collection and use of information to make decisions about 
programme, product, process or system (Cronbach, 1972). 

♦ Evaluation is the process of ascertaining the decision areas of concern, selecting 
appropriate information, and collecting and analyzing information useful to decision- 
makers in selecting choices among alternatives (Alkin, 1975). 

The common features of the preceding definitions of programme evaluation may be 

summarized as follows: 



Evaluation is the systematic process of collecting and analyzing data in 
order to determine whether and to what degree objectives have been or are 
being achieved. To evaluate is to determine the value oft something. 
Something has value if it is necessary, desirable, useful or important. It is 
also of value if it can serve a purpose or cause an effect. / 



As discussed in Chapter One, evaluation aims to answer the following questions: 

♦ To what degree has the objective been attained over time? 

♦ Is the programme cost-effective? 

♦ What impact/effect does the programme have upon the target beneficiaries? 

♦ What decision should be taken on the programme? 

A good programme evaluation should be able to generate information in order to answer 
the above questions. Further, this information “must be defensible. There must be a 
method to its collection. Thus, evaluation must be organized ... information should have 
the quality of being exact and precise . .. [and] must be usable in the improvement of some 
developmental, educational or training programme” (Bhola, 1990). The primary goal of 
evaluation is to generate information that can be used in the planning and improvement of 
current literacy and continuing education programmes as well as gain insights for future 
development efforts. 
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5.2 The Need for Programme Evaluation 

For programme managers and planners, decision making involves making a choice among 
two or more alternatives or options. To make an intelligent choice, one needs valid and 
reliable information. Evaluation provides decision makers with the information they need 
to make informed decisions about the programme. The programme planners, on the other 
hand, need evaluation data to serve as a basis for planning and strategy formulation in the 
succeeding phases of the programme or in planning programmes of similar nature in the 
future. Evaluation needs to be carried out in order to determine the extent to which goals 
and objectives are achieved and what actually takes place during the operation of the 
programme. Evaluation findings are the main basis for changes and improvements in the 
programme with focus on quality control and progress assessment. Information generated 
provides answers to questions such as why certain events do or do not occur. 

5.3 Functions of Evaluation 

Programme evaluation has several functions (Bhola, 1990): 

♦ Informational . This function provides usable information for feedback, specifically 
information that can be used to improve programme implementation. 

♦ Professional. This function aims to increase understanding about the means and ends 
of a programme, demonstrate the effectiveness or failure of programme plans and 
strategies, strengths and weaknesses, and recommend corrective actions. 

♦ Organizational. This function helps an institution to analyze its goals, objectives, 
structural strengths and weaknesses. It helps in determining organizational renewal 
and restructuring. 

♦ Political. This function consists of setting agendas and generating discussions and 
debates on certain issues and concerns. It is designed to promote group participation 
and interest among programme management, planners and administrators. 

♦ Socio-psychologicaL This function aims to give programme clients and other 
stakeholders a feeling of security. Issues and problems are simplified in order to arrive 
at conflict resolution and remedies. 

♦ Historical. The historical functions of evaluation include recording and documenting 
events and major programme activities to provide insights for future programme 
directions and actions. 



5.4 Scope and Coverage of Evaluation 

Various categorizations of evaluation are based on the information needs of the 
programme. Some common evaluation categories are as follows (Bhola, 1990): 

♦ Internal and external evaluation. The distinction between internal and external 
evaluation is in terms of who conducts the evaluation. In general, internal evaluation 
is conducted by the programme staff themselves. In instances when they use the 
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services of evaluation specialists, they are in control of the process of evaluation, from 
the planning stage up to the dissemination and use of evaluation results. External 
evaluation is conducted by evaluators who do not belong to the programme. The 
services of an outside evaluation group are engaged by the programme to help ensure 
objectivity in the evaluation process. 

♦ Formative and summative evaluation. The concepts of formative and summative 
evaluation are primarily based on process. Formative evaluation is the evaluation of 
the programme and its various components in the very process of their formation. The 
information generated can be used to improve the programme services, structure, 
systems, schedule of activities and other elements. The emphasis is on process. 
Summative evaluation is normally designed to "sum things up". It is conducted at the 
end of the programme in order to ascertain whether the objectives have been achieved 
and its impact on the clients, as well as its cost-effectiveness. The results of 
summative evaluation provide the basis for the decisions that should be taken by 
programme planners and managers. 

♦ Evaluation based on the units of analysis such as learner evaluation, programme 
evaluation, group evaluation and community impact evaluation. 

5.5 Evaluation at Various Stages of Programme Implementation 

Evaluation work commences as early as the programme conceptualization phase. It 
progresses with the increase of programme activities and delivery of services to the 
beneficiaries. The evaluation concerns at the various stages of the programme cycle are as 
follows (SEAMEO INNOTECH, 1997). 

♦ Evaluation during the pre-implementation phase During the planning and 
conceptualization of the programme, evaluation is done to obtain information about 
the quality of the product and services of the programme prior to actual programme 
implementation. The main evaluation activities at this stage are. review of the 
programme plan and materials developed; ascertaining the internal consistency among 
programme objectives, activities and resource allocation plan; and synchronizing the 
programme schedule with staff assignments and planned activities. 

Some types of evaluation conducted during this stage are: 

• Needs assessment; 

• Baseline evaluation; 

• Feasibility study; and 

• Educational policy analysis. 

Evaluation conducted during the pre-implementation phase focuses on certain 
activities related to programme design and development. 
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Pre-implementation Activities 


Evaluation Questions 


♦ Formulating goals and objectives 


♦ Are the project goals/objectives designed 
to meet the stated educational needs and 
problems? I 


| ♦ Planning the resources required 

1 ♦ Planning the activities and time 
| frame 

| ♦ Planning the personnel required 


♦ Are the resources (budget, materials, etc.) 
properly allocated for optimal use? 

♦ Are all specified activities logically con- 
sistent and adequate to bring about the 
attainment of the objectives? 

♦ Are all needed and qualified personnel \ 

recruited, with their roles/functions 
properly described? 


| ♦ Writing the trial materials. 


♦ What is the quality of the materials in 
terms of the following: 

• Internal consistency; 

• Logical sequence; 

• Comprehensiveness; 

• Relevance to user experience; \ 

• Relevance to real life situation; and 

• Comprehensibility of text. 



♦ Evaluation during the programme implementation phase. During this phase, the 
main task of evaluation is to collect evidence about the effectiveness of the programme 
or services under various conditions. 

Some types of evaluation conducted during this programme phase are: 

• Trend analysis; 

• Process evaluation; 

• Formative evaluation; and 

• Experimental study. 

Some evaluation activities and questions that need to be answered during this phase 
are: 



O 

ERIC 



61 



52 O Practitioners' Manual on Monitoring anil Evaluation 



\ Activities 


Evaluation Questions 


Prior to Full-Scale Implementation 

♦ Validating the programme or services 
with samples of the target population 


♦ Can the users follow . all prescribed j 

tasks as planned? j 

♦ What is the quality of the interim j j 

outcome? Is it what we expected? | 

♦ Is there anything difficult or problem- j 

atic which suggests some modifica- j 
tions? j 


During Full-Scale Implementation 

♦ Ascertaining the delivery of pro- ! 
| grammes and services as planned 


♦ Can the programme produce the kind j j 
of changes in clientele called for in j | 
the objectives, or specified after the | 
materials have been used? 

♦ Should the services be implemented or j 

terminated? \ j 

♦ Is there anything that should be j| 

revised? ! 



♦ Evaluation at the programme completion phase. During the programme completion 
stage, the results and effects of services and products may be evaluated through the 
following types of studies: 

• Summative evaluation 

• Cost-benefit analysis 

The evaluation activities and questions that need to be answered during this phase 
include: 



Activities 


Evaluation Questions 


i ♦ Institutionalizing the innovation after 
the materials have been revised and 
| disseminated to the proper authority. 

♦ Nationwide implementation may be 
| expected. 


♦ How effective are the overall out- ; 
comes or effects of the project? What 1 
is its impact? 

♦ How reasonable is the cost-benefit Ij 

ratio? ■ jj 

♦ Should the programme be terminated? ij 

modified? Or continued? jj 

j ♦ How sustainable is the programme? 
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5.6 The Process of Developing an Evaluation Plan 

Developing an evaluation plan for a project is a process of thinking out in advance the 
procedure to be followed and the resources to be employed, human and material, in order 
to obtain the information that is needed to appraise the worth or value of a programme 
(SEAMEO INNOTECH, 1997). The process involves identifying and describing in 
advance the data to be collected, the procedures to be used in gathering and processing the 
data, and a plan for the preparation and submission of reports. The result of the process is 
an action plan that will guide the evaluator in organizing, directing and controlling 
resources and activities so that the evaluation objectives will be achieved. The preparation 
of an evaluation plan should be a co-operative activity of the planners, clients, sponsors 
and evaluators of the project. 

Figure 5. 1 presents the steps in preparing a programme evaluation plan. 



Figure 5.1 Steps in Preparing an Evaluation Plan 




Step One: State the objectives and indicators. Just as in programme monitoring, the 

evaluation plan should have objectives that are specific, measurable, 
achievable, result-oriented and time-bound. 

The objectives usually include the indicators and the programme 
evaluator’s main task is to identify them. When the objective does not 
have explicit indicators, then the indicators have to be stated clearly in 
measurable terms. 
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Step Two: 



Step Three: 



Step Four: 



Step Five: 



Step Six: 



Step Seven: 



Step Eight: 



Step Nine: 



Develop the evaluation design. An evaluation design is a plan that 
dictates when and from whom information will be gathered during the 
course of the evaluation. The first and obvious reason for using a design is 
to ensure a well-organized evaluation study: the right people will take part 
in the evaluation at the right time. A design, however, accomplishes for 
the evaluation something more useful them just keeping data collection on 
schedule. A design is basically a way of gathering comparative 
information so that results from the programme being evaluated can be 
placed in a context forjudging their contribution and worth. 

Prepare a sampling strategy for making the evaluation. A sampling 
strategy is drawn when the programme to be evaluated is a large one with 
many participants at diverse sites. The evaluator will have to take a 
sample of the people to be tested, interviewed or observed. The sites may 
also be assessed. 

Describe the kinds of data to be gathered. The objectives of the 
evaluation should determine the kind of data to be gathered. For example, 
if the objective is to determine whether or not an innovation can indeed 
significantly increase pupil achievement in mathematics, then it is 
necessary to obtain achievement scores in mathematics for two com- 
parable groups. 

Select and plan the construction/purchase of data-gathering instruments. 
There are numerous evaluation instruments that may be used in collecting 
data at the field level. Some of them are readily available in research 
institutions and may be purchased in order to save time and effort. 

Plan the collection of data. Some of the questions to be answered in 
planning the data collection mechanism are: 

♦ Who will collect the data? 

♦ When and from whom will'data be collected? 

♦ How will the data be collected? 

Plan the data analysis. The evaluation objectives and the indicators will 
provide guidance in determining the statistical treatment of the data. 

Estimate the cost of the evaluation. The preparation of the evaluation 
budget is an important component especially if the evaluation activities 
will be financed from the programme budget. From the beginning, 
programme evaluators should be aware of the financial constraints on the 
services they provide. 

Plan the mechanism for reporting and disseminating evaluation results. 
As early as the planning stage, the programme evaluator should determine 
what kind of report to submit and who shall receive the report. 
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5.7 Preparing a Logical Framework for a Project to be Evaluated 

The essential information in the evaluation plan may be condensed into a sample table 
which presents a logical framework of the evaluation in a highly visual form. 

The logical framework can serve several purposes. It gives the evaluator a total view of 
the entire evaluation job to be done. Without reviewing the entire evaluation proposal, he 
or she can at once have a visual picture of the components of the evaluation, the major 
activities to be performed and their meaningful relationship to one another. 

The following is an example of a logical framework. 



Table 5.1 A Part of the Logical Framework for the Evaluation 
of the Social Development Programme of the 
Zamboanga del Sur Development Project 



1 Areas of 
! Evalua- 
tion 


Evaluation 

Question 


Indica- 

tors 


Evalua- 

tion 

Design 

Sample 

Scheme 


Statis- 

tical 

Analysis 


Data 
to be 
Gathered 


Instru- 
ment 
to be 
Used 


Data 

Collec- 

tors 


Schedule j 
for 
Data 

Gather- j 
ing 


V— 

; Nutrition 


Is there 


Changes 


One group 


Partial 


Measure- 


Child 


Develop- 


Every ; 




improve- 


in height 


pre- 


correlation 


ments 


develop- 


ment 


1 5 days : 




ment in 


and 


post-test 


to control 


height and 


ment 


team 






nutritional 


changes 


design 


the effect 


weight of 


record 


service 


| 




status of 
children as a 
result of the . 
social 

development 

programme? 


in weight 




of 

maturation 

Compari- 
son of 
results 
with 

height and 
weight > 
norms 


children 
3-6 years 
old 




members 

for 

health 
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Chapter Six 

IDENTIFYING AREAS OF EVALUATION 



This chapter aims to familiarize practitioners of 
literacy and continuing education to enrich their 
competencies in identifying evaluation areas in 
literacy and continuing education programmes. 
Specifically, they will learn about: 

* major areas of evaluation; 

* variables for evaluating literacy and continuing 
education; and 

* guidelines for identifying evaluation areas. 
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6.1 The Need for Identifying Evaluation Areas 

Identifying evaluation areas largely depends upon the purpose of evaluation. There cannot 
be a single set of criteria that would prescribe the areas to be considered for the evaluation 
of literacy and continuing education programmes. Even if two projects with similar 
objectives were taken for evaluation, the areas of evaluation would normally differ. For 
example, if the purpose of evaluation for one project is to assess the current mode of 
operation in order to further improve the system, the major thrust of the evaluation would 
be the assessment of inputs and the process of programme operations. In the case of 
another project, the purpose might be to know the end results of the project and its impact 
on the lives of the participants. In such a case, the evaluators would focus more on the 
accomplishments and outcomes of the project. 

Although there are variations in areas of evaluation, some areas are commonly covered. 
The variations here are in degree of emphasis or focus on certain areas. In the following 
section, major areas of evaluation will be discussed. 

6.2 Four Major Areas of Evaluation 

Taking into account the focus and content of the evaluation process, we can identify four 
evaluation areas: context evaluation, input evaluation, process evaluation, and output 
evaluation. The following table defines the scope of these evaluation areas: 



Table 6.1 Major Evaluation Areas 





I Areas 


c f 

Scope | 


j Context 


Relative environment, conditions under which the literacy project is 
implemented, needs and opportunities, problems, concurrent systems ji 


j Input 


Availability and use of resources, designs for project implementation, 
strategies applied, input supply system | 


j Process 


Procedural records, defects and difficulties in providing inputs j 


| Output 


Output from the project, attainment of targets and objectives, 
continuation of output ji 



Within the broader scope of evaluation under each area, there can be a number of options 
to cover. Selection of the options depends on the purpose and type of evaluation. Certain 
common options are mentioned below, classified into various evaluation areas. 

Context Evaluation 

National education policy 
Literacy situation in the community 
Socio-economic strata of the people 
Background of the project 



♦ 

♦ 

♦ 

♦ 
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♦ Project objectives 

♦ National/organizational setting of the nonformal education system 

♦ Implementation plan of the project 

♦ Cultural context 

Input Evaluation 

♦ Organization of learning centre/physical infrastructure 

♦ Teacher profiles 

♦ Training of teachers 

♦ Quality of project personnel 

♦ Relevancy of curriculum 

♦ Teaching-learning materials (texts and supplementary aids, both print and nonprint) 

♦ Equipment used 

♦ Types of supervision provided 

♦ Adequacy of budget 

Process Evaluation 

♦ Teaching-learning process 

♦ Community participation 

♦ Local resource mobilization 

♦ Linkage with other rural development agencies 

♦ Supervision process 

♦ Monitoring and feedback system 

♦ Human resource management 

♦ Logistics administration 

♦ Financial management 

Output Evaluation 

♦ Coverage of people 

♦ Types of people covered 

♦ Type of literacy skills acquired 

♦ Coverage of people in occupational education 

A descriptive classification of variables for evaluation (Bhola, 1990) is shown in the 
following table. 
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Table 6.2 Variables for Evaluation of Literacy Programmes 



p Variables 


Variations/Options 


Inputs 


j Teachers 


Particular education levels and social classes 

Extent of field-work experience 

Level of commitment to development work 

Teaching competence and teaching experience j 

Direct appointment versus secondment from a parent department 
Continuity versus turnover j 

Workload of teachers j j 


j Learners 


Educational background 
Social class and value orientations 
Commitment to development 

Motivation to learn j j 


| Teaching-learning 
| facilities 


Teaching materials: quantity, diversity, quality 1 

Materials and facilities 

Indigenous versus imported materials j 

Instructional and duplication equipment j 

Characteristics of learning sites 


| Process 


; Instructional/ 
j Infonnational 


Conceptualization of teaching as knowledge transfer, skills training, ; 

behaviour modification, socialization* etc. 

Integrated versus discipline-oriented curriculum development (i.e., i 
instructional organization) 

Teaching and learning styles 
Substantive knowledge versus process emphasis 
Presence versus absence of curriculum validation through needs j 
assessment j 

Availability and non-availability of counseling and guidance services i 


1 Organizational/ 

I Structural 

j Distributive/ 

I Maintenance-related 


Organizational health status j 

Organizational capacity rating 1 

Quality of administrative support 
Co-ordination with extension staff and services 


j| Context 


j Organizational 


Organization culture 

Institutional relationship (horizontal and vertical) with other organ- j 
izations j 


! i Environmental 


Surroundings 

General social climate in the country 


j Outputs 


Literate adults at various literacy levels 

Trained development workers with various competencies 

Emergent role identities 

Differential experiences of trainers \ 

Quality of radio programmes i 
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6.3 Other Areas of Evaluation 

Outcome evaluation . The evaluation of output is designed to determine the immediate 
effects of the project on the lives of the participants or in the community. At this stage, the 
immediate results from the project’s output are measured in qualitative terms. 

Impact evaluation . This evaluation effort focuses on long-term changes that have taken 
place in the lives of the people or in the community as a result of the project 
implementation, it consists of comparing the situation after the project was implemented 
with the situation prevailing before the project began. The impact can be on the lives of 
individual participants or on the community as a whole. 

Cost-benefit analysis. This is conducted to determine the unit costs of the project for 
achieving results. Money spent for providing literacy or continuing education and benefits 
derived from this expenditure at individual and national levels are analyzed through this 
evaluation. The analysis may relate to the output, outcome or impact of the project, 
depending on the purpose of the evaluation Costing is also used as an important indicator 
for deciding further expansion or replication of the project. 

Continuation . The implication of a literacy and continuing education project is no doubt 
far-reaching. But what is equally important for measuring the success of any project is to 
see to what extent the output or outcome of the project is sustainable. Will there be any 
risk of neoliterates becoming illiterate again or other project benefits dwindling away 
because of the lack of follow-up activities? In addition, it is important to see whether 
certain project activities (a community awareness media campaign, for example) continue 
to have an impact even after the project is completed 

Equity . The equal participation of people from various levels in the literacy and 
continuing education programmes is very important for measuring the impact of the 
project on the community. The participation of women or people from socially depressed 
communities, for example, is an important area of evaluation. 

6.4 Guidelines for Identifying Evaluation Areas 

In order to ensure effective evaluation, it is essential to identify specific areas of evaluation 
before developing the evaluation plan. The measurable indicators for each area should 
also be developed. The indicators help in drawing up specific questions to be answered in 
the evaluation. Further, they should reflect the expectations of the project manager or the 
person requesting the evaluation. Here are some guidelines for identifying evaluation 
areas: 

♦ Study the objectives and expected outcomes of the literacy programme. The best 
strategy is to focus on clearly defined programme objectives. There is a need to break 
down these objectives into concrete small targets or elements. 

♦ Identify variables for measurement of performance. The variables should cover all 
major areas of evaluation, i.e., context, input, process and output. 
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♦ Analyze the variables within the broad project framework and identify specific queries 
to be addressed during the evaluation. Formulate questions for the measurement of the 
variables and work out the possible evaluation indicators. 

♦ Develop a matrix of variables. It would show the relationship between the objectives 
of the project and the variables set for evaluation. It would be a set of terms of 
reference for evaluation planners that would guide them in designing the evaluation 
scheme. 



Table 6.3 A Sample Matrix Showing the Relationship between 
Project Objectives and Evaluation Variables 



Project Objectives 


Areas of Evaluation 


] Provide basic literacy to all 
] illiterate adults 15-45 years 
! old 


■ Graduation rate 

■ Socio-economic background of learners 

■ Participation rate of women 

■ Level of literacy attained 

■ Changes in literacy rate in the area 

■ Effects of literacy on the life of the people and the 
community 


I Improve the skills of 
| learners in community water 
i management 


■ Extent to w'hich the participants acquire the skills to 
handle the water problem locally 

■ Types of skills acquired 

■ Are skills transferable and sustainable 

■ Whether there is scope for further improvement of ! 
skills 

■ Are the acquired skills in use 

■ Effects resulting from skills development 


! Provide competencies for 
|j practical application of 
| literacy skills in relation to 
| meeting basic needs of 
i living 


■ Focus on training in functional literacy and de- 
emphasizing literacy learning for its own sake or as 
an independent activity 

■ Integration of literacy training with production 
based/income generation activities 
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Chapter Seven 



DEVELOPING THE EVALUATION DESIGN AND 
MANAGING THE EVALUATION WORK 



This . chapter aims to enable the planners and 
practitioners of literacy and continuing education 
programmes to: 

* gain skills in developing the various components of the 
evaluation design, including: the conceptual framework 
of the evaluation, methodology, tools /instruments, 
sampling techniques, data collection, analysis of data, 
roles and responsibilities, costs and training; and 

* understand the processes involved in managing 
evaluation work. 
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7.1 The Evaluation Design 

After identifying the areas of evaluation, the next task is the development of the evaluation 
design. In general, the goal of an evaluation effort is to assess programme performance 
when it begins, during implementation, and when it is over. An evaluation design covers 
the following: 

♦ objectives of the evaluation; 

♦ methodology; 

♦ tools/instruments; 

♦ sampling; 

♦ data collection; 

♦ analysis of data; 

♦ roles and responsibilities of various persons involved in the evaluation; 

♦ cost/budget; and 

♦ orientation/training of evaluators. 

The evaluation should be planned and conducted in anticipation of the needs of various 
interest groups, so that their co-operation may be obtained and possible attempts by any of 
these groups to interfere with the evaluation work may be prevented. Some questions that 
need to be addressed in preparing the evaluation design are: 

♦ What existing data will be used and how will they be obtained? 

♦ Who will collect new data? 

♦ Who will have oversight and management responsibility for data collection? 

♦ How often will data be collected? 

♦ How often will the data be reported? 

♦ Will data be gathered from all programme participants or only a sample? 

♦ If there is a sample, how will it be selected? 

♦ How will findings be reported? To whom? In what format? When? How often? 

♦ How will results be used? 

Programme evaluators should make an outline of an evaluation design which is the same 
for all types of evaluation, whether context, input, process or output and impact 
evaluation. 

The following are some examples of evaluation designs: 

Example 1 - Evaluation of Learning Outcomes: Learner Evaluation 

In literacy and continuing education programmes, one of the areas for serious 
consideration is the literacy level or competencies of the learners who participate in 
literacy or continuing education classes. The procedures of learner evaluation are as 
follows: 
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1. The goals or objectives of the literacy and continuing education programme should be 
clearly understood. The programme is a means to change the programme beneficiary 
from a passive to an active learner. The learning focus includes not only literacy 
skills, but also the importance of problem-solving skills and social awareness. The 
broad objectives of the programme may be: 

♦ to develop literacy and numeracy skills and apply these skills in everyday life; and 

♦ to determine the indicators of literacy skills, namely: 

reading skills: Read correctly simple passages of about 5-6 sentences in a minute, or 
approximately 10-20 words and figures (1-1,000). 

writing skills: Copy a minimum of 10 words per minute, take down dictation at a 
speed of at least 7 words per minute. 

numeracy skills: Calculate 1-3 digit figures involving simple addition, subtraction, 
multiplication and division. Demonstrate a practical knowledge of metric weights and 
measures, as well as multiplication tables up to 10. 

application of literacy skills: Read simple printed texts, such as captions, signboards, 
posters or newspaper; write simple letters; fill in forms; follow instructions on labels 
for fertilizers, pesticides, medicines, etc. 

2. Design the evaluation plan including time frame and cost. There are four phases in 
learner evaluation. 

Initial phase. This phase identifies the literacy level of the learner at the time he or 
she joins the programme. The evaluator can use word-picture cards or simple word 
cards. 

Concurrent. This phase is carried out on a continuous, ongoing basis for diagnostic 
or improvement purposes. This evaluation can be based on the lessons being taught. 

Terminal evaluation. This phase will be carried out at the completion of the 
programme to find out the degree of learning competencies gained by the learner at 
that time. 

Evaluation of literacy retention. This phase will determine the degree to which the 
learner has retained literacy skills after programme completion. 

3. Choose the sampling strategy. 

4. Design and select the tools and techniques for data collection. Learner evaluation can 
be both formal and informal. It becomes formal when questionnaires and performance 
tests are designed and used to assess the progress made by learners. It is informal if 
the methods used are semi-structured and flexible. There are various methods that can 
be used for measurement, namely interview, analysis of records/notebooks, observa- 
tion, or group discussion. 

5. Analyze the data after gathering the information about learner performance, namely 
the scores of literacy skills tests, information about using literacy skills in daily life 
and other needed data. 
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6. Prepare a report informing the learner about his or her literacy skills level. When 
learners receive feedback about their progress, they will be encouraged to improve 
their literacy skills and will have a higher motivation to participate in the literacy 
programme. Additionally, the evaluation results can serve as a basis for expanding the 
programme to new clients. If any weak points are identified, new strategies and 
approaches may be devised. 

Example 2 — Evaluation Design on the Improvement of Instruction 

At the beginning, several questions should be asked to identify problems with instruction 

and to suggest revisions. For example: 

♦ What are your biggest problems with the instruction? 

♦ If you could change one thing in this lesson, what would it be? 

♦ Are there any sections that seem too difficult? Too easy? 

♦ Do you have any suggestions for improving the materials? 

♦ Are there any topics that require more explanation? More practice? 

♦ Do the examples relate to your real-world experiences? 

The questions may focus on specific areas of intervention or perceptions of learners. 

Examples are given in Table 7. 1 . 



Table 7.1 Samples of Questions with Specific Focus 



Environment 


Worth 


1 

| Could you study this at home or in 
class? 

Could you use this without help? 


Would this learning help you do your 

job better? | 

What new topics did you learn? 


Interest and Acceptance 


Perceptual Aspects 


Did you feel bored or challenged? 
Would you use this? 


Can you easily see the screen? jj 

Can you hear everything? 


Revision - General 


Revision - Specific 


| If you could change one thing, what 
would it be? 

§ Who do you think could use this? 


How would you redesign the feedback? j 

What should the menu contain? | 



Evaluation Design and Managing the Evaluation Work O 67 



Table 7.2 shows an example of a specific evaluation instrument for recording learner 
comments. 



Table 7.2 Learner Comments 




Unit 1 

Learner comments 
Page/screen 



Evaluator questions 
Questions 



Aspects to observe/record 
Aspect 



Comments: 



Answers 



Debriefing questions 
Questions 



Unanticipated outcomes and comments 



Observation/performance 



Answer 




7.2 Evaluation Designs Which Include the Four Areas of Evaluation 

A matrix of an evaluation design which includes the four areas of evaluation discussed in 
Chapter Six is as follows: 
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Table 7.3 Example of an Evaluation Design 
with the Four Areas of Evaluation 



Context 


Input 


Process 


Output/Outcome 


| Environment 
criteria/ 


Resources 


Activities 


Changes, results 


indicators 


Learner data 
Staff/instructor data 


Learner tasks and 
interaction 


Learner data [ 

Staff/instructor 




Financial data 
Curriculum data 
Other data 


Staff/instructor tasks 
and interaction 
Extra-setting tasks/ 
interaction 
Other data 


data 

Financial data 
Curriculum data 
Other data 


Environmental 

constraints 


Resource 
constraints 
(resource provision) 


Process constraints 


Environmental 

constraints 


Environmental 

constraints 


- Input constraints 


- Process constraints 


Output constraints 
Learner objectives 
Instructor 
objectives 
Other objectives 



The following case study presents the processes and time frame for evaluating the 
development and implementation of a functional literacy curriculum in Thailand. 



An Evaluation of the Development and Implementation of a 
Functional Literacy Curriculum in Thailand 

The Northern Regional Nonformal Education Centre in Thailand carried out a project 
known as The Development of a Functional Literacy Curricular Programme in 
Northern Thailand, 

The project objectives were to: 

♦ revise the earlier functional literacy curriculum and learning materials; 

♦ train regional and provincial non-formal education personnel in curriculum 
development and project management; 

♦ improve training models, strategies and training manuals for volunteer teachers and 
related personnel; and 
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♦ create a working network between the regional and provincial non-formal education 
centres. 

An evaluation design was developed in order to assess project performance vis-a-vis its 
objectives, as well as to determine the strengths and weaknesses of the project’s procedural 
design and implementation. The following table describes the case study design. 



Programme 

Objectives 


What is Evaluated 


Evaluation 

Objectives 


Evaluation 

Methods 


Time 

Required 


I Evaluate the 
j models and ■ 
j strategies in the 
j development and 
j management of the 
j functional literacy 
j curriculum and 
j learning materials 


CONTEXT 


The relevant 
environment of the 
ongoing programme, 
i.e. 

* the curriculum and 
learning materials 
and their 
performance; 

* the effectiveness 
of programme 
management 


Assess to what extent 
the current curriculum 
and learning materials 
are congruent with 
today’s social, cultural 
and economic 
changes. 

Determine the 
effectiveness of 
programme 
management as well 
as problems occurring 
in its implementation 


Document analysis 
(reports of research 
meetings and expert 
interviews) j 

Comparison of 
content presented in 
the materials with 
current conditions 
in the society 1 


6 months 


INPUT f 


The inputs required 
in the development 
of the curriculum 
materials and the 
programme 
implementation 

1 


Consider the 
objectives of the new 
programme in terms of 
its importance and 
responsiveness to the 
problems found in the 
context evaluation 

Analyze the potential 
costs and the benefits 
of one or more 
procedural designs 
and select the most 
suitable and feasible 
design. 


Document analysis 

Organization study 
(manpower 
resources, budget 
allocation, 
management 
system, etc.) 

Interviews with 
experts in 
programme 
planning and 
methodology, 
curriculum and 
materials 
development 

Analyze 
possible 
alternatives and 
their costs 


3-4 

months j 



Source: ATLP. 



6 KJ 



best copy available 
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Programme 

Objectives 



What is 
Evaluated 



Evaluation 

Objectives 



Evaluation 

Methods 




PROCESS 



Programme 

implementation 

* Activities 

* Programme 
management 



up on 

implementation of 
the curriculum 



Monitor the programme 

Assess the programme 
activities in terms of its 
objectives, procedures 
and effectiveness 

Determine the strengths 
and the weaknesses of 
the implementation and 
provide information for 
remedial decisions 

Note: 

Programme activities 
will include. 

Workshop to develop 
curriculum and 
materials 

Training of volunteer 
teachers 



Interview/ 
questionnaires 
(respondents are 
programme staff/ 
administrators, 
teacher trainers, 
volunteer teachers 
and other involved 
personnel) 

The interview 
should focus on: 

* Expected 
processes 

* Expected quality 

* Actual processes 
and quality 

Compare the 
expected and 
actual outcomes 

Identify the gaps 
and problems and 
propose alternative 
solutions 



j PRODUCT 






The programme 


Assess programme 


Achievement tests 


product 


success 


Interviews 


The immediate 


Evaluate learner 


Questionnaires 

Qualitative 


programme 


achievements and skills 


evaluation 


I achievements 


and assess the 


approaches 


: The programme 
achievements in 
! the long run 


acquisition of desired 
behaviours according to 
the programme 
objectives 


Case studies 



Detennine the 
effectiveness of 
the curriculum, 
learning materials 
and the curriculum 
management 

Study the 

programme impact 
on learners and 
their communities 



The programme 
impact 



Assess the effectiveness 
of the programme 
during the 

implementation period 
and after the project is 
completed 

Determine the 
programme impact 
on learners and their 
communities 
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7.3 Sampling 

If the project to be evaluated is a large one with many participants at diverse sites, it will 
not be easy to evaluate everyone or to look at every site. With much effort, the evaluator 
can reach each participant with a highly-focused questionnaire, containing a few or no 
open-ended questions. This would enable the evaluator to gain an overview of perceptions, 
but might not provide in-depth data on the subject matter. If the evaluator needs more 
insights into a particular group’s perceptions, he or she will need to select a sample. 

A sample is a portion or subset of a larger group called a population. The target 
population consists of the institutions, persons, problems and systems to which or to 
whom the evaluation findings are to be applied or generalized. 

Some examples are:* 

Example 1 

Target population : All teacher training programmes in the district or province. 

Programme: Continuous Quality Improvement: An intervention to monitor and change 
the quality of teacher training. One index of quality is the performance of students on 
reading and math tests administered throughout the district or province. 

Sample: Five teacher-training institutions will be selected to try out the Continuous 

Quality Improvement experiment. After 1 year, for all participating teacher trainees, a 10 
per cent sample of student performance in reading and math will be taken. 

Comment: This evaluation aims to evaluate all teacher-training programmes in the 
province. Five will be selected for a Continuous Quality Improvement programme. To 
appraise the program’s quality, the evaluator will sample 10 per cent of the students to 
assess their performance in reading and math. The findings are to be applied to all 
teacher-training programmes in the province. 



Example 2 

Target population: All students needing remediation in reading 
Programme: Options for Learning 

Sample: Five schools in three districts; within each school, 15 classes; for each class, at 
least 20 students who need remediation in reading 

Comment: Students who need assistance in reading are the targets of the programme. The 
evaluators will select five schools in three districts, and within them, 15 classes with at 
least 20 students in each. The findings are to be applied to all students who need special 
aid in reading. 



* Arlene Fink, Evaluation for Education and Psychology, California: Sage Publications, 1 995. 



o 

ERLC 



80 



72 O Practitioners' Manual on Monitoring and Evaluation 



The sample results provide estimates of what the result might be if the whole population 
was surveyed. Therefore, there is need to select a sample very carefully. 

There are several different methods used to select a sample and the choice of 
method depends on the circumstances. The most common sampling strategies are: 

Probability Sampling 

This is considered the best way to ensure the validity of any inferences made about 
programme effectiveness. In probability sampling, every member of the target population 
has a probability of being included in the sample. 

There are four kinds of probability sampling. 

1 . Simple random sampling. In simple random sampling, every subject or unit has an 
equal chance of being selected. Because of this equality of opportunity, random 
samples are considered relatively unbiased. This technique makes use of a table of 
random numbers, a lottery procedure or the rolling of a die. 



Example: There are six teachers. We want to know with a 
reasonable degree of accuracy how they are using teaching aids. 
We do not have resources to monitor all six of them and decide 
to monitor only two. How many different pairs of teachers could 
be picked out if there were six of them? There are the following 
possibilities, lettering them A to F inclusive. 

AB AC AD AE AF 

BC BD BE BF 

CD CE CF 

DE DF 

EF 

There are 15 different examples we can choose. Write each of 
these 15 examples on a slip of paper and put all 15 into a box. 
Pull out a slip. Each slip has an equal possibility of being 
picked, i.e.,'l in 15, a random sample. 



As the population size in evaluation is usually large, the number of different samples is 
also large. If the above sampling method is used, it will be too laborious. Statisticians 
have prepared a Random Number Table which can be found in practically all 
statistics books. With the help of this random table, a sample can be selected without 
any bias. 
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2. Systematic sampling. This method can be used in selecting a sample when the 
population is finite or when the random sampling technique may be too laborious to go 
through. An example of this method is as follows: 

Example: An evaluator had a list of names of 3,000 students from 
] which a sample of 500 was to be selected. In systematic sampling, 
divide 3,000 by 500. 

] | 3,000 : 500 = 6 Every sixth name would beselected. 

An alternative would be to select a number by random, say by tossing a 
die. Suppose a toss came up with the number five, then the 5th name 
would be selected first, then the 11th name, 17th and so on until 500 
! names are selected. 



3. Stratified random sampling. In this sampling technique, the population is 
divided into subgroups or strata and random sampling is used in each stratum. 
This helps in getting a more accurate representation of the sample. Examples 
of such strata are geographical location, age group, gender, and income level. 

4. Cluster sampling. This is a variation of the simple random sampling technique 
and is particularly appropriate when the population is infinite, when the list of 
members of a population does not exist or when the geographic distribution of 
subjects is widely scattered. 

Non-probability or Convenience Sampling 

1. Convenience sampling. In this method, samples are selected because they are 
available. Some members have a chance to be chosen, while others dp not. As a 
result, the data collected from a convenience sample may not represent all the members 
of the target population. However, in some cases, appropriate statistical analysis may 
be used to make the convenience sample a representation of the target population. 

2. Deliberate or purposive sampling. In this sampling technique, the sample is selected 
for some specific reason. 

There is no fixed number or percentage of subjects that determines the size of an adequate 
sample. It may depend upon the nature of the population, objective of the study, or kind of 
data to be gathered and analyzed However, the sample should be representative of the 
population. Normally, when the sample is larger, the more representative it will be of the 
population But unnecessarily large samples will not be good samples The best sample is 
that which is both economical and representative. 
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The following table can be used for determining sample size: 



Table 7.4 Table for Determining Sample Size from a Given Population 



N 


s 


N 


s 


N 


s I 


10 


10 


220 


140 


1,200 


291 1 


15 


14 


230 


144 


1,300 


297 J 


20 


19 


240 


148 


1,400 


302 1 


25 


24 


250 


152 


1,500 


306 ' jj 


30 


28 


260 


155 


1,600 


310 | 














35 


32 


270 


159 


1,700 


313 | 


: 

40 


36 


280 


162 


1,800 


| 317 ! 


| . 45 


40 


290 


165 


1,900 


| 320 j 


50 


44 


300 


169 


2,000 


322 


55 


48 


320 


175 


2,200 


327 














60 


52 


340 


181 


2,400 


331 || 


65 


56 


360 


186 


2,600 


j 335 | 


| 70 


59 


380 


191 


2,800 


1 338 | 


| 75 ' 


63 


400 


196 


3,000 


| 341 


80 


66 


420 


201 


3,500 


346 | 


j 












i 85 


70 


440 


205 


4,000 


351 j 


90 


73 


460 


210 


4,500 


354 | 


95 


76 


480 


214 


5,000 


! * 357 - | 


100 


80 


500 


217 


6,000 I 


361 1 


1 10 


86 


550 


226 


7,000 


364 | 














120 


92 


600 


234 


8,000 


367 


130 


.97 


650 


242 


9,000 


368 ; 


140 


103 


700 


248 


10,000 


370 


150 


108 


750 


254 


15,000 


375 


160 


113 


800 


260 


20,000 


377 














170 


118 


850 


265 


30,000 


379 


180 


123 


900 


269 


40,000 


380 ! 


190 


127 


950 


274 


50,000 | 


381 j 


200 


132 


1,000 


278 


75,000 


. 382 | 


210 


136 


1,100 


285 


100,000 j 


384 



(N= Population size; S = Sample size) 



Source: H.S. Bhola, UNESCO Institute for Education German Foundation for International Development. 
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7.4 Evaluation Tools and Techniques 

The tools discussed in Chapter Three can likewise used as programme evaluation tools and 
instruments. In addition, this chapter focuses on techniques that may be used in the 
evaluation work at the learner's level, village level and district level. They can be 
quantitative or qualitative; a combination of the fools will help in cross-checking and in 
confirming information gathered. 

A non-formal education programme evaluation requires informal, flexible, diversified 
methods of assessing growth and change. Assessment should be done systematically and 
carefully recorded in order to draw objective conclusions for eventual corrections and 
modifications in the programme. Some commonly used evaluation tools and techniques 
are: 

Techniques Tools 

Assessment test 
Observation guide 
Interview guide 
Structured interview guide 
Diary, learner’s record and 
audio-video tools 
Questionnaire, checklist 

Test. Classroom tests are necessary to check the learning achievements of learners. They 
are useful in helping learners to know their progress in learning. They can be either 
performance or assessment tests. Both are useful in making learners aware of their 
performance. 

In assessment tests, the questions should be practical and applicable to the daily lives of 
the learners. Some samples are in Appendix III. 

Initial evaluation: 

■ Word-picture cards 

■ Simple word cards 

■ Writing words and figures on blackboard 

■ Matching words and pictures. 

Periodic progress evaluation: 

■ Group discussions 

■ Interviews 

■ Checklist items 

Final evaluation: 

■ How many sales did you make last week? 

■ What foods do your children like? 



Test 

Observation 

Interview 

Community meetings 
Documentation 

Survey 



7 6 Practitioners' Manual on Monitoring and Evaluation 



In a performance test, specific skills are measured. The leamer/participant is given a task. 
The evaluator then assesses the quality of the performance or final product. A 
performance test requires trained observers/evaluators to ensure a fair appraisal of 
performance. This type of evaluation can be made more interesting by holding an 
exhibition upon graduation, or competitions within a district or at the national level. This 
activity helps to boost participant morale; the participant who is successful could serve as 
a model for other learners. 

Observation. Observation is the seeing and listening at the actual place of a project, such 
as the community learning centre. It is very necessary especially at the learner’s level 
where behaviour in the local environment, changes in attitude, and interactions in the 
community are some of the evidence of progress. 

The purpose of the observation may be known or not by those observed and it may be. 
structured, unstructured or open-ended. It provides an opportunity to collect first-hand 
information which was not anticipated before the evaluation. Information can be gathered 
by taking notes, photographs and video. 

It is very important that evaluators agree on the focus and theme before going to the field 
After gathering the information, they will have to discuss the observations before they 
draw conclusions. This method consumes a lot of time and labour, and the evaluators can 
influence the environment in the sense that the learners can act differently when they know 
they are being watched 

Interview. Interviews may range from informal and conversational to closed and highly 
structured They allow for two-sided interactions and are very useful in collecting 
information from someone who has difficulty in reading. This method provides an 
opportunity for participants to give appropriate answers and for interviewers to learn 
something about the learner’s growth, interests and development. 

Although it provides the richest source of reliable information in the shortest time, it 
requires special skills in asking questions and in interpreting responses that are off the 
record. It is normally done together with other means of gathering information An 
example of an interview guide is in Appendix III. 

During the administration of the interview, the evaluator should: 

♦ Observe how the learner reacts to the problems discussed in the class; 

♦ Be sensitive/alert to learner attitudes toward the acceptance of new ideas; 

♦ Analyze learner responses and questions; and 

♦ Look for changes in behaviour. 

Community meeting. This is a structured assembly of a large group of community 
residents and leaders that provides a forum for sharing and disseminating information 
(planning, monitoring and evaluating projects/activities), as well as making recommenda- 
tions and decisions. In an evaluation exercise, a community meeting opens the process to 
more people and makes them committed to the project, thus leading to a greater co- 
operation especially during the collection of data. 
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It is best to inform the community of the meeting before the actual date. The meeting has 
to be well structured and planned to ensure that the objectives of the meeting are achieved. 
The meeting should start with a statement of objectives It should be brief and geared to 
the concerns of those attending the meeting. When necessary, the meeting can be divided 
into small group discussions and a plenary session for sharing findings. 

Documentation. Documentation may be in print or nonprint forms. 

Diaries and learner records. A diary is a daily record of events that occur over a period 
of time. The record should focus on specific activities and the “what, who, when, why, 
and how” for these activities. Recording has to be done right at the beginning of the 
project. In keeping a group record, a member of the group has to be assigned to update the 
data. 

The data in these records are hard evidence of programme progress and development The 
information can be used by the leamers/participants for their self-evaluation or by an 
outside evaluator such as the project supervisor at the village or district level 

The data are useful for monitoring and, eventually, for evaluation of the programme, 
especially in identifying major problem areas and noting changes and accomplishments. 
The documents in themselves can be learning tools for community members. 

Audio-video tools. Audio-video tools can vividly document programme progress, 
outcomes, processes, activities and impact on participants and their communities. They 
carry emotional and intellectual messages and may reveal aspects of a project not 
generated in structured interviews or questionnaires. Audio-videcr tools can be used either 
to document aspects of a project or stimulate analysis related to interpreting project 
problems and impact. Events are recorded at the beginning of the project, during project 
implementation and after the project is over. 

Audio-video documentation may be done by the participants themselves and used to 
identify and prioritize problems, as well as to set goals and determine ways to improve 
project activities. Audio-video tools should be used to cross-check and confirm the 
information gathered from other evaluation tools, such as community meetings and learner 
records. 

Survey. Surveys are popularly used in baseline data collection. Information can be 
gathered through checklists or structured questionnaires. Surveys are inexpensive to 
administer and allow for more privacy on the part of the participants Tabulation is easy 
but generation of responses is usually slow and respondents must be literate. 

7.5 Collection of Data 

Choosing the method. In every type of evaluation, some type of information- gathering 
techniques and processes are involved. There may be need for some observation, 
questioning, testing and interviewing. The most appropriate method of data collection 
should be selected based on the following questions. 
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♦ Will this method provide the needed information? 

♦ Will this method be acceptable to the people involved? 

♦ Does time permit the application of the method and the necessary analysis of the data? 

♦ Are enough resources available to use this method? 

Managing the collection of data. Data collection may be quite difficult if not managed 

properly. While collecting data, the following guidelines should be followed. 

♦ Collect only a moderate amount of data. 

♦ Devise economical methods of gathering information; whenever applicable, use 
existing resources. 

♦ Collect data in different ways for validation purposes. 

♦ Collect data from different samples for a more realistic representation of the evalua- 
tion respondents. 

♦ Develop a cordial relationship with the respondents. Building rapport is a very 
important part of data collection. Inform them of the objective of the study and collect 
data in a participatory manner. 

♦ Before going to the field for collection of data, make sure that there are no festivals, 
weddings or important occasions on the day that actual evaluation is to be conducted. 
Moreover, natural calamities and other emergency situations may disturb the data 
collection process. 

♦ Develop cordial relationships with the important persons in the field. They will help in 
introducing the evaluators to the people and explaining the purpose of the visit. 

♦ If male data collectors get data from women, sometimes it may be difficult to generate 
accurate or honest responses. Therefore, the evaluation team should be composed of 
male and female members or an acceptable community member should be chosen to 
accompany the evaluators. 

♦ The rules of conduct during the interview or field-work should be properly understood 
and observed by everyone in the data collection group. 

♦ To generate the needed information, sometimes the evaluators may have to stay in the 
field for many days or have to make several trips to the field. 

♦ Investigators who are assigned the task of data collection should be trained properly in 
the techniques of data collection before they go to the field. They should also be 
trained in administering the evaluation instruments. 

♦ Usually, it is not easy to collect data from the pre-selected sample and the investigators 
have to make substitutions for the unavailable respondents or are forced to make do 
with smaller samples. In such cases, the investigator must keep an honest record of 
changes made in the sample so that appropriate judgement can be made at the time of 
interpreting the results. 

♦ Instruments should be pretested before they are administered to the intended sample. 
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♦ Completed questionnaires and other records should be handled carefully. Data are 
precious and should be treated as such. The investigators should be given clear 
instructions about the system of returning completed questionnaires. 

♦ Completed questionnaires and other forms should be marked and coded as soon as 
received. 

7.6 Analysis of Data 

Once the data have been collected, the evaluator must analyze the findings and present 
them with recommendations for programme improvement. The presentation is usually in 
printed form but can be enriched by audio-video materials. Data analysis is a critical 
process because it leads directly to the formulation of conclusions and decisions by the 
programme planners and managers. 

There are two components, namely data processing and data analysis. These two terms 
have substantial overlaps. Data processing makes the data ready for data analysis. 

Data processing and data analysis can consist of the following activities: 

♦ Categorize the data according to: 

■ gender 

■ occupation 

■ age 

■ group 

■ educational background 

■ other 

♦ Conduct a quantitative analysis using: 

■ descriptive mean 

■ percentage 

■ mean (averages) 

■ standard deviation 

♦ Conduct a deductive analysis with appropriate statistical procedures, such as 

■ z-test 

■ t-test 

■ chi-square test 

■ multiple regression test 

■ others 

Data analysis using statistical procedures helps to ensure more accuracy than relying on 
manual computations. Further, there is existing computer software to help in more speedy 
data analysis. For statistical formulas and other applications, programme evaluators can 
refer to any standard textbook on statistical analysis. 

The basic strategy in data analysis is to classify the collected information, first with one 
variable and subsequently with several variables in terms of frequencies, cross tabulations, 
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and the calculation of basic statistical parameters. Statistical analysis makes use of 
indicators such as the mean, mode, median, variance and standard deviation. 

Usually, once the data have been processed, they are analyzed with the following 
purposes kept in mind (Bhola, 1990): 

♦ Relating data to preconditions and entry behaviours; 

♦ Viewing data in context; 

♦ Relating data to what is already known; 

♦ Looking for correlations and causations; 

♦ Re-examining the assumptions made in the design of the evaluation; and 

♦ Looking at the limitations of the data. 

The data analysis techniques in Chapter Three of this manual can also be used in 
the analysis activities of programme evaluation. 

7.7 Evaluation Schedule 

If sufficient time is not provided to carry out key tasks or if it is scheduled when the people 
who need to be involved are heavily committed elsewhere, the evaluation cannot be 
conducted properly. An essential part of the planning process, therefore, involves the 
preparation of a realistic implementation schedule for the evaluation work. Sometimes, 
evaluation schedules become weak when: 

♦ The evaluation plan is not discussed properly and in detail with the people whose co- 
operation is necessary for carrying out the task. It usually results in misunderstanding 
and, consequently, diminished commitment. 

♦ Meetings are arranged when people are not available. 

♦ Not enough time is given for reflection upon key issues, gathering necessary 
information, or taking action. 

♦ Plans are not made according to the availability of secretarial or support staff. 

♦ Insufficient time is allowed for respondents to return questionnaires and for non- 
respondents to be followed up. 

♦ Over-ambitious interview schedules are arranged with a large number of interviews 
squeezed into a short period of time. 

♦ Inadequate time is given to data gathering. 

♦ Inadequate time is provided for analyzing the data. 

Effective scheduling needs to take the following into account: 

. Realistic estimates of time required to complete tasks. 

Availability of the members of the evaluation team. 

Sufficient time allowed at key stages in the process for taking stock of the issues and, 
if necessary, replanning the evaluation design. 



♦ 

♦ 

♦ 
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Table 7.5 gives an example of a data-gathering and processing schedule. 



Table 7.5: Sample of a Data Gathering and Processing Schedule 



Activity 


Estimated time 


Decide needed information 


2 weeks j 


Decide data-gathering method 


0.5 weeks i 


Decide evaluation method 


0.5 weeks . j 


j Determine sample 


0.5 weeks I 


j Design interview schedule 


2 weeks ||! 


; j Inform respondents/ar range 
j interviews 


1 week j|| 


j Brief interviewers 


1 week . !ji 


Conduct interviews 


3 weeks jj 


If Process data 


4 weeks | 


: i Write report 


4 weeks | 


Type and reproduce report 


2 weeks i]j 


Circulate report 


’ 1 week ! 


! Review report 


2 weeks | 


Plan structure of feedback seminar 


6 weeks |i 


Plan content of feedback seminar 


3 weeks III 


| ! Advertise feedback seminar 


3 weeks ill 


! Run seminar 


0.5 weeks ill 



Based on: Managing Evaluation in Education, Routledge, 1 992. 

If activities are realistically scheduled and people work with commitment, evaluation 
activities have substantial chances to succeed. However, scheduling should be dealt with 
more formally, especially in cases of complicated or large-scale evaluation. 

7.8 Roles and Responsibilities 

Management of evaluation is getting things done with and through others. It involves 
recruiting people who possess the knowledge, skills and commitment in order to achieve 
the desired evaluation objectives. This may involve managing a carefully selected team of 
people who will work together on day-to-day basis, or it may involve a team of people with 
whom the evaluator is already working. In either case, the nature of their involvement has 
to be well-planned. The degree of their involvement may vary at the various stages of 
evaluation. 

The following concerns need to be addressed: 




90 



82 <* Practitioners' Manual on Monitoring and Evaluation 



♦ Who is to lead the process and what should this involve? 

♦ Who else should participate in the management process? 

♦ To whom should those managing the process be accountable and how? 

♦ To what extent will support be required from certain people? 

Evaluation is a value-laden activity. While involving individuals and groups in the various 
aspects of evaluation, this concern should always be kept in mind A major dimension of 
evaluation is soliciting the commitment of the programme management to take action as a 
result of the insights gained The more open, shared, consultative and involving the 
process, the more possible it is to solicit commitment. Table 7 6 and Table 7.7 provide a 
framework for considering this issue and developing plans for the management of the 
evaluation activity. 



Table 7.6 Individual/Group Involvement in Evaluation 




Table 7.7 A Sample Evaluation Process 



Stage 1 : Determining the focus 

Who should be involved 


How? 






Stage 2: Preparing plans 

Who should be involved 


How? 






| Stage 3: Clarifying purpose 
| Who should be involved 


How? 






Stage 4: Determining indicators 
Who should be involved 


Dow? 






Stage 5: Doing sampling 

Who should be involved 


How? 






Stage 6: Managing data gathering 
Who should be involved 


How? 






Stage 7: Analyzing data and reporting 
| Who should be involved 


How? 







Source: Managing Evaluation in Education, Routledge, 1992. 
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7.9 Staff Orientation and Training 

Orientation and training of personnel to be involved in evaluation are essential. These 
activities involve more than a cursory introduction to the evaluation manual. The training 
design for them should be developed properly, with attention given to the following: 

♦ Training should be more skills-oriented. 

♦ Collaborative planning should be carried out by those who will co-ordinate the 
orientation and training. 

♦ Training should involve more than just introduction of the evaluation manual. 

♦ Sufficient instructional materials should be made available to the trainees. 

♦ Workshop, demonstration, role playing and simulation exercises should be the major 
training strategies. 

♦ Trainees should learn how to write evaluation proposals, develop evaluation 
instruments, collect data, deal with respondents, make inferences, probe, draw insights, 
and make the appropriate recommendations. 

♦ Data collection, processing and analysis should also be major topics during the 
orientation/training activities. 

Evaluation personnel should be well versed with various evaluation techniques. In general, 
all personnel who are involved in the planning and management of evaluation should 
acquire the relevant knowledge and skills to do it. They should acquire specific 
competencies for the purpose, including: 

♦ A clear understanding of the concept, objectives, scope, coverage and other pertinent 
information related to evaluation. 

♦ Basic techniques of evaluation. 

♦ Ability to write an evaluation proposal. 

♦ Specific evaluation techniques to assess leamer/neo-literate achievement level. 

♦ Evaluation techniques to assess the various activities of literacy and continuing 
education programmes. 

♦ Evaluation techniques to assess the strengths and weaknesses of the project and the 
factors responsible for them. 

♦ Techniques of data collection, processing and analysis. 

♦ Report writing and dissemination strategies. 

The duration of training generally depends on the background and previous experience of 
the evaluation staff. Moreover, it would depend upon their degree of involvement in the 
evaluation process. 

Training of data collectors . In most cases. Level B or Level C personnel will not be able 
to collect all data single-handedly. They will need the assistance of colleagues or other 
field workers for collecting and analyzing data. These field investigators must be 
adequately trained in the techniques of administering evaluation instruments and in 
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analyzing the evaluation results. They should undergo a well-planned orientation/training 
programme in order to adequately prepare them for the evaluation work. 

7.10 Evaluation Costs 

Managing the costs of evaluation involves essentially three stages: 

♦ identifying the resources required; 

♦ assessing the costs involved in obtaining these resources; and 

♦ ensuring that these costs will be met (budgeting). 

Some of the resources needed to carry out an evaluation activity are: 

♦ people (managers, field staff, analysts, etc ); 

♦ various kinds of equipment, materials/services (e g., designing/printing costs and 
postage for sending questionnaires or memos about a meeting or group discussion, 
computers, duplication, etc ); 

♦ space for meetings, work stations; and 

♦ vehicles and transportation expenses. 

Some of these resources require additional financial programme expenditure. Cost 
analysis serves as a basis for budgeting. While budgeting, it is necessary to assess not 
only what resources are needed but also the strategies to ensure that these. will be available 
at the right time. 
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Chapter Eight 

REPORTING AND USE OF EVALUATION RESULTS 



This chapter aims to familiarize practitioners in 
literacy and continuing education programmes with 
strategies on reporting and using evaluation results, 
specifically: 

* basic considerations in reporting evaluation findings; 

* importance of and factors affecting, use of these 
fndings; and 

* Database Management System. 



o 

ERLC 



94 



86 $ Practitioners' Manual on Monitoring and Evaluation 



8.1 Basic Considerations in Reporting Evaluation Findings 

Pertinent data collected and analyzed need to be reported to groups or individuals who are 
concerned with the project at different structural and functional levels. Some basic 
considerations should be taken into account: 

♦ The results of the evaluation must be communicated in a form and manner that are 
understandable to the intended users. Technical and statistical terms should be 
avoided or carefully defined. 

♦ The recipients of the results should be given specific recommendations and 
suggestions. Anyone involved with or concerned about the evaluation results should 
be informed of both the appropriate and inappropriate uses of the results. 

♦ The various concerned users have a right to know the results of evaluation. The 
evaluator’s responsibility is to report the results and to help users to understand what 
they mean. 

♦ The findings should be targeted to user interests at different levels. 

The ultimate objective of reporting evaluation findings is to ensure that the results serve as 
a basis for making informed decisions on whether to maintain, expand, scale down, or stop 
the programme intervention The evaluation findings, conclusions and recommendations 
provide the policy makers and programme decision makers with the necessary information 
for drawing up plans of action 

8.2 Reporting Formats 

To make an effective presentation of evaluation results, it is useful to keep in mind the 
various forms of reporting. Although it may be in print or non-print form, the most 
common form of reporting is a written report. It serves at least two purposes: 

♦ To provide an opportunity for the evaluators to organize their thinking, to draw 
conclusions, to weigh and consider the implications of the evaluation activities. 

♦ To act as an instrument of communication between evaluators and audiences. 

The essential objective of a written evaluation report is to make a comprehensive record of 
the evaluation activities, including the background and the context of the areas of 
evaluation; the evaluation process; evaluation design and tools; the results obtained; and 
the conclusions reached. 

In general, an evaluation report should have the following main parts and sections: 

title page 
abstract 

general background 

the evaluation questions asked/other evaluation instruments 
assumptions 

procedures and methods used 
evaluation design 



♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 
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♦ conceptual analysis and definitions 

♦ standards and criteria of success 

♦ sample and units of response 

♦ tools and instruments 

♦ field-work procedures/processes/timetable 

♦ findings 

♦ discussion of results 

♦ further evaluation and feedback needs 

♦ bibliography 

♦ appendices 

An example of the contents of an evaluation report is as follows (Bhola, 1990): 

The Implementation and Effects of the Kenyan Literacy Programme 

1 . Research design and implementation 

■ Background 

■ Objectives of the study 

■ Research design 

■ Research implementation 

• Selection of locations 

• Selection of respondents 

❖ The literacy learners 

❖ The control group 

• Preparation of research instruments 

❖ The learners’ questionnaire 

❖ The teachers’ questionnaire 

❖ The literacy test 

• Data collection 

• Data analysis 

2 The Kenyan literacy programme 

■ The national context 

■ The national literacy programme 

■ Literacy work before 1979 

• The national literacy programme of 1 979 
3. The location profiles 

8 Geographical features 

■ Population 

■ Economic activities 

■ Socio-cultural characteristics 

■ Socio-economic services 
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• Self-help and local development 

• School education 

• The literacy programme 

• Summary 

4. The literacy centres: characteristics and functioning 

• Buildings used for literacy teaching 

• Original use 

• Materials used for the construction of literacy classrooms 
❖ Conditions of the teaching environments 

• Teacher characteristics 

• Categories 

• Gender 

• Age and marital status of teachers 

• Teachers’ educational qualifications 

• Teachers’ experience 

• Previous occupation of teachers 

• Second occupation 

• Ties with the local community 

• Teachers’ attitudes and job satisfaction 

• Teaching/leaming aids 

• Teaching aids 

• Learning aids 

• Records 

• Learning exposure 

• Content 

• Class projects 

• Guest lecturers 

• Centre committees 

• Average attendance 

• Summary 

5. General characteristics of the literacy learners 

• Gender 

• Age 

• Marital status 

• Number of children 

• Languages spoken by the learners 

• Mother tongue 

• Ability to speak Kiswahili and English 
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■ Learners’ religion 

■ Learners’ occupations 

■ Summary 

6. Learners’ home environments 

■ Type of housing 

■ Possession of books and magazines 

■ Availability of audio-visual equipment 

■ Exposure to the mass media 

■ Summary 

7. Educational experiences of learners 
Exposure to primary schooling 

Literacy classes before obtaining proficiency certificates 

Reasons for joining the literacy programme and benefits derived from it 

Duration and regularity of literacy class attendance 

Learning after the literacy certificate 

■ What adults would like to learn after the literacy certificate 

■ Duration of literacy class attendance after the certificate 

■ Participation in other types of course 

■ Listening to the special DAI radio programme 

■ Summary 

8. Using literacy and numeracy skills 

■ Differences between locations 

• Reading 

• Writing 

• Calculating 

■ Other types of difference 

• Gender 

• Speaking Kiswahili 
° Summary 

9. Functional knowledge, attitudes and practices 

■ Measuring functional knowledge, attitudes and practices 

■ Differences between the literates and illiterates 

• Knowledge 

• Attitudes 

• Behaviour 
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■ Differences among the literates 

• Location 

• Other differences (gender, age, year of certificate, primary school attendance 
and literacy class experience) 

■ Summary 

10. Literacy and numeracy skills acquired 

Grading test results and setting performance standards (numeracy, reading, writing) 

■ Results for the whole sample (numeracy, reading, writing, relations between the 
three types of skills, global results) 

■ Differences between locations 

■ Other differences (gender, learning expenence, year of literacy certificate) 

■ Summary 

1 1 . Conclusions 

■ The learners and their motivation 

■ The functioning of the literacy programme 

■ The effects of the literacy programme 

A good evaluation report should contain the most important information necessary for a 
reader to understand the whole programme and the evaluation work. In other words, the 
audience should know exactly what was done and how, using what samples and what 
instruments, and what conclusions were drawn. The reader should also be able to draw 
independent and alternative conclusions based on the collected data. 

8.3 Target Groups of Evaluation Reports 

It is necessary to develop a plan for the dissemination of the evaluation findings. A central 
concern is to carefully study the audience’s interests and needs. It is a fact that different 
publics have different concerns and levels of sophistication when it comes to understanding 
data. For example, the Board of Education at the district level could be less interested in 
the details of the evaluation experience but could be more interested in changes and 
improvements of the programme. Thus it is important to differentiate users and the 
purposes for which data are to be used. The figure below shows an example. 
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Figure 8.1 Levels and Types of Users and the Purposes 
for which Data Are To Be Used 



USERS 



PURPOSE 
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8.4 Factors Affecting the Use of Evaluation Results 

Evaluation is the systematic process of collecting and analyzing data in order to make 
decisions. Thus the results of evaluation can be used by a variety of groups or individuals 
for making decisions concerning the effectiveness and efficiency of a literacy or a 
continuing education programme. There is a considerable concern among evaluators about 
evaluation results being ignored. How useful results are obviously depends to a great 
extent on how they are used. If the results are viewed as being valuable sources of input 
and are studied carefully, they can lead to improved decision making at all levels. 

In reality, there are many factors that can affect the use of evaluation results, including the 
following: 

♦ Pre-existing evaluation constraints. Even before an evaluation begins to take place, 
some events already roughly influence the scope and direction of the evaluation. 

♦ Orientation of the users. The interests and attitudes of the users of evaluation are 
central. Potential users of evaluation results have many varied questions or concerns 
about the programme being evaluated. Thus the effective path to utilization is to 
produce information about matters of clear interest or concerns to users. 

♦ Evaluator's approach. Each evaluator has his or her own evaluation styles and 
methods. The approach could exert a strong and guiding influence on the evaluation. 

♦ Information content and reporting. A written report may be laden with evaluation 
information such as quantitative data, statistical analyses and conclusions about the 
programme. But there may be multiple evaluation reports, not simply a single report. 
There are many presentations that can also be considered reports in the sense of 
disseminating information. Therefore, the format in which the information is presented 
is also a significant influencing factor. 

8.5 Database Management System 

In order to give the users easy access to evaluation results and to help them use the 
evaluation information effectively, it is necessary to build a Database Management System 
(DBMS). The DBMS is responsible for the organized input, storage and retrieval of data. 
The role of a DBMS is described in Figure 8.2. Other pertinent information on an EMIS 
is discussed in Chapter Four of this manual. 
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Figure 8.2 Role of the Database Management System 
within the General Structure of an EMIS 
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SAMPLE FORMS FOR DATA COLLECTION 
IN MONITORING 



A. SAMPLE SURVEY QUESTIONNAIRE 



The Literacy Coordinating Council (LCC) is currently conducting a mid-term evaluation 
of the Comprehensive Education and Community Development Programme (CECDP) - 
Valencia Project. The overall objectives are to know the status of the project and to 
identify factors which may affect its success and sustainability. One of our evaluation 
methodologies is this survey questionnaire. 

You have been selected as one of the respondents of our survey. We will appreciate it if 
you can take time out to complete this questionnaire. Your answers will be kept strictly 
confidential. 

I. Socio-demographic Profile 

1 . Names (optional): ’ 

2. Age: 

3. Sex: □ M □ F 

4. Occupation (title/designation): . 

5 . Average monthly income: ■ 

6. Length of residence in the area: ; 

II. Development of Programme Continuum 

1 . What is your official designation in the CECDP? 



2. Who appointed/designated you to the CECDP? 
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3. How many months have you been involved in the project? 

Q 1 to 3 mos. 

Q 4 to 6 mos. 

Q 7 to 10 mos. 

Q 1 1 mos. - above 

4. Please put a check {•/ ) mark before the statements which best describe the objectives 
of the project? 

Q increase agricultural productivity and income of rural farmers/families 
Q promote Valencia as a tourist destination 
Q reduce illiteracy 

Q serve as extension community (social laboratory) of the UP system 
Q serve as pilot area for educational innovations 
Q provide livelihood and entrepreneurship training 
Q set up barangay learning centers 

Q develop social consciousness among the people of Valencia 
Q organize community organizations 

5. Do you think the programme addresses the needs of Valencia? 

Q Yes Q No G 1 don’t know 

5.1 What are some of the priority needs not addressed by the project, if any? 



6. CECDP is a model in “integrated” development programmes. Which of the following 
statements best describes an integrated approach? (Please choose one (1) only) 

Q The project has many components 

Q The project covers the entire community 

Q The project is linked with the national/municipal development plan 
Q The project has various interrelated components 
Q The project is parts of the Social Reform Agenda (SRA) 
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III. Programme Clientele, Requirements and Resources 

1 . Do community members participate in the project? 

Q Yes (Please proceed to No. 2) Q No (Please proceed to No. 3) 

Q I don’t know 

2. How do they participate? 

Q attend (planning) meetings 
Q give financial support 
Q give free labour 
Q give equipment/facilities/materials 

(please specify ) 

Q participate in surveys 

Q attend trainings/seminars 

Q join organizations formed by the project 

Q others, please specify ; 



3. In what other ways can community members participate in project activities? (Aside 
from those mentioned above) 



4. What sectors of the community can still be mobilized to participate and support the 



project? 




□ Youth 


Q Professionals 


Q Women 


Q Business/Industry 


Q Church/Religion 


Q Academe/Schools 


□ Local NGOs/POs 


Q Local mass media 


Development of Literacy/Training Materials 

Are you familiar with the learning modules produced for this project? 


□ Yes 


□ No 
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2 . If yes, what are the subject matters/topics of these modules? (Please enumerate) 



3. Please put a check (/) mark before the statements which accurately describe the 
module development process. 

□ The modules are relevant to the needs of the community 

□ The modules are easy to read and understand 

□ The module lessons are practical (easy to apply) 

□ Intended beneficiaries use the module 

□ External agencies helped prepare the module 

□ The modules were written by local experts 

□ The modules can have a long life span 

V. Organization and Management 

1. Please put a check (/) mark to the statement which best describes the project’s 
organizational structure. 

□ structure is still evolving 

• □ well-defined with clear functions and responsibilities 

□ not defined at all 

2. How would you describe the planning and decision-making process? 

□ top-down (centralized) 

□ participatory (decentralized) 

3. What are the roles/contributions of the following agencies to the project? 

Literacy Co-ordinating Council 
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Local Government Unit (Municipality of Valencia) 



DECS - Division of Negros Oriental 



University of the Philippines System 



4. Does the project provide you an opportunity to leam/acquire new knowledge/skills? 

□ Yes , d No d I don’t know 

5 . If yes, in what areas? 

[-1 Community Development 
d Rapid Community Assessment 
d Monitoring and evaluation 
d Planning 
d Networking 

d Conducting trainings/seminars/meetings 
d Writing modules 

d Social (community) mobilization and advocacy 
d Integrated Development 
d Social Reform Agenda 

d Technology (in agriculture, environment, tourism, health, education, etc.) 
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□ Others, please specify 



6. Do you think CECDP is sustainable? 

Q Yes □ No □ I don’t know 

7. What strategies can you suggest to ensure the sustainability of the project? 



Thank you for your co-operation. 
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B. WEEKLY SELF-EVALUATION SHEET 1ST WEEK 



WEAVING 

SKILLS 

I DEVELOPMENT 


Excellent 


Good 


Average 


Fair 


Poor 


1 Natta 










/ 


1 Somchit 












I Pimpa 


s 










1 Sukon 












[Nipa 




S 






- 


| Sakorn 












lUbon 




S 






] 


| Prapim 












1 Saengdee 












1 Supa 


y 










[Payao 




S 








i Vanida 










✓ 


i Tanom 












| Surat 










/ 


| Porjal 












i Srinuan 








V 


— 


[Watana 








S 




| Pratoom 












\ Nongnuj 








S 




j Total 


2 


4 


i 


4 


8 



Source: Patchanee Natpracha and Alexandra Stephens. Taking Hold of Rural Life. FAO of the U.N. 
Regional Office for Asia and the Pacific (RAPA) Bangkok 1 990. 
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C. PROBLEM STUDY QUESTIONNAIRE 



The Problem Study Questionnaire included here was designed for use with women in rural 
areas and is reproduced in M. Bazany, Work-Oriented Adult Literacy in Iran: An 

Experiment, (Final Technical Report), Vol. Ill, Part IV. 2, Work-Oriented Adult Literacy 
Project (WO ALP) Esfahan/Iran 1973. 

The instrument was designed by the Iran project for use in interviews to collect 
information on vocational and specific related interests of rural women to be the basis for 
preparing need-based materials for use in family planning programmes. 



Source: Josef Muller and Anja Dietrich; Dossier of Evaluation Instruments for Literacy Programmes 
Center for Education Science and Documentation , Bonn Federal Republic of Germany 1989. 



Rural Areas 

1. Name of the interviewed: First name: 

2. Age: 

3. Marital status: Q married Q single Q divorced Q widowed 

4. Number of children: Q males Q females 

5. Literacy: Q can sign his name Qcan read the Quoran 

Q read a little Q read and write a little Q read and write 

6. Main occupation: 

7. Where do you work (location): 

8. How many hours per day do you work in this job: 

9. Do you have any other work besides this job: Q yes Q no 

If yes, what is it. • 

10. Where is it (location)? 

1 1 . How many hours per day do you work in this job: ' 

12. Are you interested in participating in any literacy class: Q yes Q no 

13 . Do you think you will be able to spare time to attend such a literacy class: 

Q yes Q no 

14. If interested, what time of day would be most suitable for you: 
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15. If besides literacy some technical (vocational) skills would be taught, what skills 
would you be interested in: (e g. modem agricultural techniques, cattle husbandry, 
poultry, welding etc.) 

16. For what reasons would you like to leam these skills (the skills that the interviewee has 

indicated): ■ 

17. Do you think there is any other job in which you might earn more money: 

□ yes □ no 

18. If yes, what type of job is it: 

19. Do you plan to change your present job in the future: □, yes □ no 

20. If yes, what type of job would you go to (take up): * 

21. Why do you want to change your present job? 



22. If some classes for men would be opened in your village, do you think the villagers 

would be interested: □ yes □ no □ don’t know 

23. If some classes for women should be opened in your village, do you think the women 

would be interested: □ yes □ no 

24. What kind of skills do you think should be taught to the villagers: 

For men. For women: 

If interviewee is married. 

25. How many of your children are going to school . 

If answer is “none”, why: ' 

26. Is your wife □ literate d illiterate 

27. If your wife is not literate, are you interested in sending her to a literacy class: 

□ yes □ no Why? 



28. If besides literacy, some skills are taught, what skills do you think your wife should 
leam: 



29. Do you think your wife herself would be interested in learning literacy combined with 

practical skills: □ yes □ no 

If interviewee is a farmer: 

30. How much land do you own (in hectares): 
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3 1 . How many plots do you have: 

32. How much land do you actually cultivate: 

33. How often do you usually plant and harvest per year: 

34. What crops are you growing: 

Type Amount of hectares used 

1 . 

2. 

3. • 

4. 

5. 

6 . 

35. Where do you sell your harvest? : 
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D. MONTHLY REPORT 



by Supervisor 

(To be filled by the Supervisor and sent to the Project Officer) 



Month ending: Year: 

Name of Supervisor: 

1. Number of instructors/inistructresses at the, end of the month: Q Male Q Female 

2. Instructors/instructresses who dropped out during the month 

Name of instructor/instructress Reasons 



I 3 . Classes which discontinued during the month 



Name of Class 


m |f 


MF 


• Reasons 











































i 4. Report on Attendance 

I Comments: Action taken: Proposals for remedial action: 




5. Materials needed (please specify): 



6. Instructors received Honorarium up to the month of: 

Comments: 

7. Any other comments: 



Date: 



Signature of Supervisor: 
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by Instructor 

(To be completed by Instructor and submitted to Supervisor) 



1 . At the end of the month of: Year: 

2. Village: 

3. Name of Instructor: 

4. Group name 

5. Number of learners at the end of the month: Q Male Q Female 

6. Total number of lessons held per month: 

7. Number of learners who absented the whole month. Q Male Q Female 

8. Number of learners who discontinued: Q Male Q Female 

9. Average monthly attendance. 

10. Equipment required: 

11. Number of days supervisor visited class: 

Number of.days extension agents visited class: 

a) Agriculture extension agent. 

b) Health extension agent: 

c) Home economics worker: 

d) Others (name): 

12. Last lesson taught at the end of the month: 

1 3 . What was done to reduce learner drop-out? What was done to bring back drop-outs to 

the class? 

14. Comments: 

15. When was the last Honoraria paid: L 

16. Literacy Committee Chairman’s comments 

17. Chairman’s Signature: 

18. Instructors Signature: 



Date: 

Date: 
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SAMPLE FORMS FOR REPORTING INFORMATION 
ON MONITORING 



A. MONTHLY REPORT 



Province and district: Year. Instructor: 

Month: Supervisor: 

Group No.: 

********************** 



A Statistics 

a) Number of sessions planned, held: 

b) Number of registrations in the batch: 

Number additional registrations: 

Number of registrations at the end of the month: 



c) Number of absences by sex and age. 



Men 


Women 


Total | 


i 

i 15-24 | 25-45 


-45 


15-24 


25-45 ! - 45 


| 


1 | 

i \ 

L, _ _JL ~ _ 








! 

.... „ i; 



d) Number of absences by cause: 



] w 1 


H 


T 


F 


D 


C I N 


! Total I 








i 




t 

s 

\ 


Ij 

; J 



Cause-of-absence code: W 

H 

T 

F 

D 

C 

N 



Work- 

Health 

Travel, emigration, seasonal migration 
Traditional holidays or festivals 
Domestic reasons, children, etc. 
Climate, bad weather 
Unidentified, etc. 
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B. Visits by Supervisor 

During the month (dates): 

Total number of visits since the beginning of the courses: 

C. Teaching Material Received 

Individual (type and quantity): • 

Collective (type and quantity): 

For instructor’s use (type and quantity): . 

Audio-visual (type and quantity): 

D. Assistance Received 

Foodstuffs (quantity): 

Overtime: ' 

Working hours: 

Fertilizers, pesticides, seeds, plants, etc.: 

Other: , 

E. Budget 

a) Salaries: 

b) Bills settled (maintenance, neating, electricity, cleaning materials, etc.) 
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. MONTHLY SUMMARY REPORTS 



Project Name: 

Project Location: Month: Sep./97 



L.D. 1 . Average Attendance of Learners in Class 















Programmi 












i Average 

Attendance 


Pre-Primary j 


Early 


1 Second 


Adolescent 


Adult 


AH 









Primary 


j Chance 










Programmes 




no 


% 1 


no 


% 


no 


% 


no 


% 


no 


% 


no 


% 


I 80% and above 


0 


0.0 j 


0 


0.0 


0 


0.0 


23 


92.0 


52 


69.3 ; 


75 


75.0 | 


! 70 - 79 % 


0 


0.0 j 


0 


0.0 


‘ 0 


0.0 


2 


8.0 


20 


26.7 


22 


22.0 


! 60 - 69 % 


0 


0.0 | 


0 


0.0 


0 j 


0.0 


0 


0.0 


1 


1.3 


1 


TO ; 


j 50 - 59 % 


0 


0.0 I 


0 


0.0 


0 


j o.o 


0 


0.0 


■ 0 


0.0 


0 


0.0 ; 


j Below 50% 


o 


0.0 ! 


0 


0.0 


°l 


0.0 

J 


0 


0.0 


2 


2.7 


2 


2.0 


j Total 


0 


0.0 j 


0 


0.0 | 


» ! 


t 

0.0 1 


25 


100.6 


75 


100.0 | 


100 


100.0. 


j Average 








j 




i 

1 

J 








i 




AAA ^ A4 i; 


3 Attendance 








j 

i 


j 22.1 

1 


20.2 J 




20.7 ji 



• 




BEST COPY AVAILABLE 




121 




112 O Practitioners' Manual on Monitoring and Evaluation 



Project Name: 

Project Location: : Month: June/97 



L.D.3. Grades Achieved by Learners 





Programme jj 


Learner 

Grades 


Pre-Primary 


Early 

Primary 


Second 

Chance 


Adolescent 


Adult 


All Programmes j| 


j 


no 


1 % 


no 


% 


no 


% i 


no 


% j 


no 


% 


no 


% | 


jj Grade A 


< 

0 ] 


| 0.0 


0 


0.0 


0 


o.o ! 


° ! 


o.o 1 


17 ! 


68.0 


17 


68.0 ; 


Grade B 


0 


1 0.0 


0 


0.0 


0 


0.0 1 


0 I 


0.0 j 


j 4 | 


16.0 


4 


160 I 


i Grade C 


0 


1 0.0 


0 


0.0 


0 


0.0 1 


0 ] 


0.0 


j 4 | 


16.0 


4 


16.0 j 


Grade D 


0 i 


j 0.0 1 


0 


0.0 


0 


0.0 


1 0 


| 0.0 j 


| 0 j 


| 0.0 


0. 


0.0 ] 


| Grade E 


0 


1 0.0 


0 


0.0 


0 


: 00 


! o 


1 0.0 


| 0 


| 0.0 


o 


0.0 | 


|; Grade F 


0 


1 0.0 ! 


0 


0.0 


0 


00 


1 0 


| O.o 1 


1 0 


1 0.0 


o i 


0.0 | 


jj Total 


o 


| 0.0 


0. 

vwvwsvwvwv 


0.0 

LWTOWAW 


0 


i o.o 


1 0 


| 0.0 


; 25 

TOWMVMW 


j 100.0 


' 25 


100.0 | 



L.D.4. Skills Achieved by Learners 





Total | 


Skills 






i Programme 


Learners 


Reading 


Writing 


Maths 


Functional 

Kn. 


Practical 
App. jj 




no j 


f | 

! \ 


no 

\ 


% j 


no 


% 


no 


| % j 


no 


; % 


no 


Pre-Primary 


0 


! 1 

0 j 


0.0 j 


0 | 


0.0 


| ; 

° | 


| 0.0 1 


0 | 


0.0 | 


0 


0.0 jj 


i Early Primary 


0 


1 0 1 


0.0 ! 


0 | 


0.0 


0 ! 


| 0.0 1 


| 0 | 


0.0 ! 


0 


0.0 jj 


; Second Chance 


0 


! 0 ] 


! o.o j 


I 0 


1 0.0 


o ! 

i 


! 0.0 


! o | 


o.o | 


0 


0.0 jj 


i Adolescent 


0 


j 0 1 

j ; 


1 00 


! o 


| 0.0 1 


0 


j 0.0 


| o | 


0.0 


j 0 


0.0 j 


Adult 


25 


j 20 


| 80.0 ! 


1 20 


i 80.0 ; 


19 | 


1 76.0 i 


! 20 j 


! 80.0 i 


| 18 


72.0 jj 


j| Total 


25 


1 20 

! 1 


S i r': ii 

Mmo^i 


|* r •- " i 

h >20' * 


j • 80:0 H 


■ . i/- - "j 

19 * 


| v 76.0 j 

1 


| 20 | 


1 80.0 i 


| 18 


72.0 jj 
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SAMPLE EVALUATION TOOLS 



A. EVALUATION INSTRUMENT 



( Excerpts from an Evaluation Instrument of 
a Literacy Programme in Malawi) 



BOM LA MALAWI 

Government of Malawi 

SUKULU ZAKWACHA 
National Adult Literacy Programme 

MAYESO A OPHUNZIRA 

Learner assessment (Tests) 



LAR/1 

Learner Assessment Reading, Section 1 

Dzina la Ophunzira 

Name of learner 

Mwamuna/Mkazi 

Male/Female 

Dzina la kalasi 

Name of centre/class 

KUWERENGA 

Reading 



(Agwiritse Ntchito m’phunzitsi pochonga mawu ndi ziganizo zowerengedwa 
molondola) This is used by the instructor when marking/ticking words and 
sentences correctly read. 
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A. 


Munda 


fetereza 


ukhondo 


chikondwerero 


nkhokwe 




garden/field 


fertilizer 


hygiene/ 


celebration/ 


grain-store 








cleanliness 


festivity 




B. 


Mwana wagwa panjinga. 









A Child has fallen from a bicycle. 

Kafumbwefumbwe waononga chimanga. 

Weevils have spoilt maize. 

C. Munthu waukhondo amayamba wasamba m’manja asanadye chilichonse 
A clean person washes hands before eating anything. 

Amatero pofuna kupewa matenda osiyanasiyana. 

He/she does so as to prevent different diseases. 



Kuwerenga ndi kuyankha mafunso 

Reading and answering questions. 

Matenda otsegula m’mimba ndioopsa makamaka kwa ana. 

Diarrhoeal diseases are very dangerous especially for children. 

Mwana atadwala matendawa tingachitepo kanthu tisanapite naye 
kuchipatala. 

When a child suffers from such diseases we can do first aid for going to hospital 
with the child. 

Njira yake nayi: Titenge lita imodzi yamadzi ophitsa. 

The method is as follows: We should have one litre of boiled water. 

Tithire suga wokwana zitsekelo za fanta kapena za kokakola zisanu ndi zitatu. 

Add 8 Fanta or Coke bottle tops full of sugar. 

Tithirenso mchere wokwana chitsekelo chimodzi. 

Also add salt one bottle top full. 

Tizimumwetsa mwanayo kawirikawiri mpaka atamaliza litayo patsiku. 

We should give the child as often as possible until the solution is finished in a day. 

Tikatero tithandiza kubwezera madzi amene akutayika postsegulapo. 

If we do this, we help the child to regain fluids lost or being lost as a result of the 
purging/diarrhoea. 

1. Kodi m’ madzi opatsa mwana wotsegula m’mimba tithiremo zinthu ziti? 

What things should we add to water which is given to a child with diarrhoea? 



2. Nanga zinthuzo zikhale zochuluka motani pa lita imodzi? 

How much should each of the things be? 



3. Nchifukwa ninji mwana otsegula m’mimba tizimpatsa madzi oterewa? 

Why should we give a child with diarrhoea such water? 
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BOMA LA MALAWI 

Government of Malawi 

SUKULU ZAKWACHA 

National Adult Literacy Programme 

MAYESO A OPHUNZIRA 

Learner assessment (Tests) 



LAR/2 

Learner Assessment Writing, Section 2 



Dzina la Ophunzira 

Name of learner 

Mwamuna/Mkazi ■ 

Male/Female 

Dzina la kalasi 

Name of centre/class 



LEMBETSO 

Dictation 
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BOMA LA MALAWI 

Government of Malawi 

SUKULU ZAKWACHA 
National Adult Literacy Programme 

MAYESO A OPHUNZIRA 

Learner assessment (Tests) 



LAR/3 

Learner Assessment Arithmetic, Section 3 



Dzina la Ophunzira 

Name of learner 

Mwamuna/M kazi 

Male/Female 

Dzina la kalasi 

Name of centre/class 



MASAMU 

Arithmetic 



9 + 5 = 


12 + 4 = 


14+16 = 


11 +22 = 


5-3 = 


15-7 = 


64-34 = 


17- 7 = 


9-5-3 = 


12-5-4 = 


tl 

00 

+ 

00 

00 


9 -s- 9 = 


3x4 = 


7x2 = 


2x1 = 


7x7 = 



1. A Phiri ali ndi mbuzi 3, ng’ombe 5 ndi nkhumba 8. 

Mr.Phiri has 3 goats, 5 cattle and 8 pigs. 

Kodi a Phiri ali ndi ziweto zingati? 

How many animals (livestock) has Mr.Phiri? 



2. Paulo chaka chatha adapeza matumba 4 a chimanga. 

Paul got/harvested 4 bags of maize last year. 

Chaka chino anathira fetereza ndipo anapeza matumba 8. 

This year he had applied fertilizer and so he has harvested 8 bags. 

Kodi chaka chino anapeza matumba angati kuposa chaka chatha? 

How many bags more than last year has he harvested this year? 
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3. A Banda adali ndi matumba a chimanga amene adagulitsa K 13.00 
thumba lililonse, mbatata matumba 6 amene adagulitsa K 2.00 thumba 
lililonse, ndiponso matumba 4 a nyemba omwe adagulitsa K 30.00 
thumba lililonse. 

Mr. Banda had bags of maize which he sold at K 13.00 per bag, 6 bags of 
potatoes which he sold at K 2.00 per bag, and also 4 bags of beans which 
he sold at K 30.00 per bag. 

a) Kodi pa matumba 10 a chimaga a Banda adapeza ndalama zingati? 
How much money did Mr. Banda make out of 10 bags of maize? • 



b) Nanga pa matumba a chimanga 10, a mbatata 6, a nyemba 4, 
adapeza ndalama zingati zonse pamodzi? 

How much money altogether did he make from 10 bags of maize, 6 
bags of potatoes, and 4 bags of beans? 



c) Akadakhala kuti a Banda sanalime mbatata akadapeza ndalama 
zingati? 

If Mr. Banda had not planted/grown potatoes, how much would be has 
got? 



d) Pa ndalama zonse zomwe adapeza adachotsapo K. 100.00 zomwe 
adakagula makasu. 

Out of all the money he realised, he used K 100.00 to buy hoes. 

Nanga zomwe zidatsala ndi zingati? 

What was his balance? 
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( Excerpts from the CECDP Evaluation Instrument) 



I. Degree to which Activities have been Completed 
According to Work Plan 






Activities in Work Plan 



Completed 

A 



Partially 

Completed 

B 



Not 

Completed 

C 



Reasons 

for 

Item C 



1 . Curriculum Development 

No. types of modules 
No. of modules prepared 
No trained 
Subject areas 
No. being utilized 

2. Resources/Manpower 
Actors/Materials 

Local gov't officials 
Barangay officials 
School officials 
Students 

3. Development, Planning and 
Organization of Management, 
Technical and Consultants 

Available development plan 
Available community profile 
Available plan of technical profile 

4. Capability Building/Training 

Availability of trained personnel 
Availability of local trainors 

5. Implementors/Beneficiaries/NGO/ 
Technical 

Local officials 
Barangay officials 
Community people 

6. Documentation/Report s/Informati on 

Progress reports 
Regular reports 
Brochures, etc. 

7. Monitoring and Evaluation 

Task Force for M and E 
Independent team 






o 
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n. Benefits derived from the Programme Activities 



1 

Is Category 


Fully 

(A) 


Partially 

(B) 

. 


No Benefit 
(C) 


Reasons ;j 
for 

Item (C) j 


j 1 . Agriculture 

j Increase productivity 

j Increased income 

j i Use of modern fanning techniques 

|i 2. Literacy 

1 Increase no. of literates 

j Increase no. of liteacy trainors 

| Increased skills 

j 3. Health and Other Social Services 

j Increased no. of mothers accessing 

I the social services 

I Increased immunization of 

j children 

j Reduction of morbidity and 

! mortality 

j 4. Eco-Tourism 

Development of new tourist sites 
Increased livelihood opportunities 
Greater sustainable development 

j 5. Actors 

j CECDP Valencia 

j LCC 

j UP-ERP 

| DECS Division OfTice 








1 1 
1 1 
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IQ. Programme Effectiveness (Degree to which Project 
Has Achieved Immediate Objectives) 



Immediate Objectives 


Fully 

Achieved 

(A) 


Partially 

Achieved 

(B) 


Not 

Achieved 

(C) 


Reasons !] 
for Item j j 

(C) ! 


J''"-' 


Progress/Perforin a nee 








| 


I 1. 


Reduction of illiteracy in Valencia by 
50 per cent 








| 


1 2. 


Availability of functional Barangay 
Learning Centres in 2 1 out of 24 
Barangays by end of 1 997 through 
Barangay cluster, etc. 










I 1 


Increase Agricultural 
Productivity and Rural 
Income in Valencia 










1 4. 


Established eco- tourism sites 










| 5. 


Promotion/development of social 
consciousness among community 











Source: Evaluation Instrument of the Comprehensive Educational Community Development 
Programme (CECDP), Valencia, Negros Oriental, Philippines. The evaluation was conducted 
by the Technical Working Group of the Literacy Co-ordinating Council of the Philippines in 
1997. 
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B. QUESTIONNAIRE: SAMPLE 



An example of a questionnaire used by an interview in a group setting. 

Group’s Name: 

Facilitator: 

Date: 



1 . Age of members: 

2. Sex of members: Q Male Q Female 

3. Is this group a church group? Q Yes Q No 

4. Has this group elected group leaders? Q Yes Q No 

4a. If yes, who are they? Q Male Q Female 

5 . When did this group start? 



6. How many members did the group have at the beginning? 

7. How many members have left? 

8. How many of the present members began with the group? 

9. How often do you meet? 



10. About how many members come to meetings? 

1 1 . Which literacy lessons have you covered? 

12. Do you have a full-time facilitator? Q Yes Q No 

12a. How often do you see him? 

12b. How much time does he spend with you? 

1 2c. What do you do with him? What do you talk about with him? 



Region: 

District: 

Village. 
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13. Do you have a volunteer facilitator? Q Yes □ No 

13 a. How often do you see him? 

13b. How much time does he spend with you? 

13c. What do you do with him? 

14. Does the Village Development Committee know about your group? G Yes G No 

14a. How do they now? 

14b. What do they think about your group? 

15. Have you begun any self-help projects? G Yes G No 

15a. What are they? 

15b. Did you finish it/them? O Yes □ No 

15c. If yes, how is it being G No G Other • 

1 5d. If not completed, why not? 

15e. Did you have any problems with this project? 

15f. Who worked on it? G Male G Female 

15g. Who thought of the idea? G Male G Female 

16. Do you have any other community project? 

17. How can you prevent malaria? 

18. How can you prevent crop disease? 

19. How much power would you say you have to improve your life? 

G much 
G quiet a bit 
G some 
G very little 
G none 

20. How do you think rural people can gain power to improve their lives? 



2 1 . How can you prevent your children from getting malnutrition? 
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22. When should you begin to give solid food to a baby? 

Q 3 mos. 

Q 6 mos. 

Q 1 year 
Q 18 mos. 

23. What do you do if your child gets diarrhea? 

24. Do you think you can improve your income? □ Yes □ No 

If yes, how? 

25. What do you think can be done to improve the condition of feeder roads? 



Prepared by 

World Education Inc. Monitoring Workshop Report, p. 54-6. 



Source: Daniel Santo Pietro (ed) Evaluation Source Book. American Council of Voluntary Agencies 

for Foreign Service Inc. Second Impression, 1983. 
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LIST OF PARTICIPANTS 


Bangladesh 


Mr. Ehsanur Rahman 
Director 

Programmes Planning and Monitoring 
Dhaka Ahsania Mission 
House No.-19, Road No. -12 (new) 
Dhanmondi Residential Area 
Dhaka- 1209 

Tel. (880-2) 815 909; 819 521-22 
Fax (880-2) 813 010; 818 522 


China 


Dr. Liu Yibing 

Associate Professor 

Deputy Director of Research Dept. 

Research & Training Center for Literacy Education 
Southwest-China Normal University 
Beibei, Chongqing 400715 
P R. China 

Tel. (023) 68252664 
Fax (023) 68867157 
E-mail: ganxun@swnu.edu.cn 


India 


Ms. Nishat Farooq 
Director 

State Resource Centre 
Jamia Millia Islamia 
Jamia Nagar 
New Delhi- 1 10026 
Tel. 6838043 
Fax 6828014 


Malaysia 


Ms. Khamsiyah bte Haji Yahya 
Community Development Division 
7th Floor, Wisma Kim Seah 
Jalan Puncak 
Off Jalan P. Ramlee 
50606 Kuala Lumpur 
Tel. 03-2326744,2326871 
Fax 03-2327646 
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Philippines 



Thailand 



Secretariat 

UNESCO/PROAP 



SEAMEO INNOTECH 



on Monitoring and Evaluation 



Dr. Zenaida T. Domingo 

Senior Specialist/Training 

SEAMEO INNOTECH 

Commonwealth Avenue, Diliman, Quezon City 

Tel. (632) 924 768 to 84 

Fax (632) 921 0224/928 7692 

E-mail, zenv@innotech.org 

Dr. (Ms.) Srisawang Leowarin 
Senior Supervisor 
Supervisor Unit 

Non-formal Education Department 
Ministry of Education 
Bangkok 10300 
Tel (66-2) 28 5151 
Fax (66-2) 2810438 



920 Sukhumvit Road 
Bangkok 10110, Thailand 
Tels. (66-2) 3910686,3910703 
Fax: (66-2) 391-0866 

Ms. Darunee Riewpituk 
Specialist in Continuing Education 

Mr. John Middleborg 

Associate Expert in Literacy and Continuing Education 

Commonwealth Avenue 
Diliman, Quezon City 
Tel. (632) 924-76-81 to 84 
Fax: (632) 921-02-24/928-76-92 

Mr Benito E. Beneza 
Ms. Erlene V. Acapulco 
Ms. Ami G. Fontanilla 
Ms. Nora D. Cabatian 
Mr. Nestor M. Laroya 



0 

_ERiC 



STCOPY AVAILABLE 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally- funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 



EFF-089 (3/2000) 





